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I.  INTRODUCTION 


A.  MOTIVATION 

In  2006  and  2007,  I  led  the  logistics  section  of  a  training  battalion  at  the 
Marine  Corps’  western  amphibious  training  base  at  Camp  Pendleton,  CA.  I  had 
just  returned  from  combat  duty  in  Iraq.  I  was  in  good  company,  because  virtually 
every  Marine  in  the  battalion  had  just  completed  one  or  more  combat  tours.  The 
Marine  Corps  used  the  training  battalion  as  a  resting  place  for  deployment-weary 
Marines.  From  the  Mayberry-like  confines  of  our  cantonment  area,  we  were 
swept  off  the  stage  of  constant  deploying,  training,  fighting,  and  standing  guard. 
Suddenly,  we  were  watching  the  war  the  way  most  everyone  else  does.  We  read 
about  the  units  whose  ranks  we  once  filled,  and  the  Marines  with  whom  we  once 
served,  as  remote  headlines  in  the  newspaper  or  on  the  evening  news.  We  no 
longer  felt  that  insight  of  understanding  the  war  from  a  first-person  perspective 
because  we  were  there  seeing  it  and  feeling  it.  Rather,  we  joined  the  millions  of 
other  people  reading  or  hearing  snippets  of  military  action,  distilled  through  a 
long  journalistic  chain.  One  did  not  have  to  read  deeply  or  listen  to  too  many 
evening  news  reports  to  gather  that  the  face  of  the  conflicts  was  ever  changing, 
casting  our  experience  quickly  and  surely  into  irrelevance.  Within  months,  our 
experiences  were  merely  anecdotal  tidbits  from  a  time  that  had  come  and  gone; 
war  stories  to  be  told  one  day,  like  those  of  World  War  II  or  Vietnam.  We  had 
been  there  and  done  that,  but  we  had  become  spectators.  Worse  yet,  our  skills 
declined.  We  were  training  recruits,  administering  the  daily  dither  of  a  home-site 
organization.  We  were  not  fighting  anyone  or  thinking  about  fighting  anyone, 
other  than  through  the  abstractions  of  firing-range  targets.  We  could  feel  our 
warfighting  skills  erode  just  as  our  experiences  melted  into  memories.  We  were 
becoming  irrelevant. 

I  had  been  in  the  command  about  six  months  when  my  assistant,  a  well- 
educated  Captain  who  had  joined  the  Marine  Corps  after  9-11  to  serve  his 
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country,  approached  me  with  a  suggestion.  He  couched  his  discussion  with  an 
overview  of  our  waning  warfighting  skills  and  a  description  of  the  hodge-podge  of 
logistics  Marines  in  our  section.  He  suggested  that  we  go  to  the  convoy  trainer. 
At  the  time,  my  knowledge  of  the  convoy  trainer  was  limited  to  some  vague 
understanding  of  simulation,  and  I  had  no  idea  where  it  was  or  who  operated  it.  I 
liked  the  Captain’s  idea,  though,  so  he  and  I  started  working  on  a  plan  when  we 
had  time,  gradually  forming  it  into  some  sort  of  executable  concept. 

First,  we  found  the  convoy  simulator.  I  Marine  Expeditionary  Force  (MEF), 
for  all  intents  and  purposes,  encompasses  all  of  the  active  duty  Marines  west  of 
the  Mississippi,  and  it  is  headquartered  near  the  beach  on  the  southern  end  of 
the  base.  Tucked  right  behind  the  headquarters  building  is  the  I  MEF  Simulation 
Center,  a  modern  box  of  a  building  partitioned  inside  to  cubicle  cells  of  computer 
activity.  We  met  with  the  staff  there,  and  they  showed  us  the  convoy  trainer, 
which  turned  out  to  be  a  bunch  of  laptop  computers  spread  through  a  few 
different  spaces.  I  had  been  in  the  Marine  Corps  long  enough  to  know  that 
anything  new  came  with  its  own  acronym  soup,  and  soon  I  was  lost  in  VBS, 
DVTE,  and  a  variety  of  other  terms.  Eventually,  I  figured  out  that  the  convoy 
trainer  was  hosted  on  Virtual  Battlespace  2  {VBS  2™),  a  networked  personal 
computer-based  first-person  shooter  game  with  civilian  roots.  The  Simulation 
Center  staff  could  use  simple  drag-and-drop  controls  to  build  any  scenario  I 
wanted  on  any  one  of  three  different  selections  of  terrain.  Quickly,  I  saw  that  we 
had  a  very  versatile  environment  with  a  setup  that  would  allow  all  of  my  Marines 
to  participate  as  individuals. 

Through  the  next  six  months,  I  worked  with  my  assistant  to  develop  and 
administer  a  series  of  classes  on  basic  convoy  concepts  and  Tactics, 
Techniques,  and  Procedures  (TTPs).  We  culminated  the  series  with  a  detailed 
higher  headquarters  operational  order  to  the  group  to  conduct  a  convoy.  We  had 
built  the  exercise  scenario  on  paper  and  provided  the  details  to  the  Simulation 
Center  staff  for  programming.  We  had  our  Marines  develop  different  plans  for 
conducting  the  mission  and  selected  a  few  of  the  best  for  practice  in  the 
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simulator.  After  all  the  preparation  work  was  complete  in  the  fall  of  2007,  the 
whole  section  went  to  the  Simulation  Center.  The  staff  there  gave  the  Marines 
about  an  hour  of  instruction.  The  first  half  was  administrative  in  nature  with 
standard  Marine  rules  for  the  obvious  such  as  keeping  drinks  away  from  the 
computers.  The  second  half,  about  30  minutes  in  length,  taught  the  Marines  how 
to  be  themselves  virtually.  Drivers  learned  to  move,  turn,  and  look  with  keyboard 
keys  and  gunners  learned  to  shoot  with  a  mouse.  Finally,  we  had  a  team  of 
virtual  warfighters,  complete  with  a  plan.  The  Marines  executed  the  mission 
once  before  lunch  and  once  after  with  comprehensive  After  Action  Reviews 
(AARs)  after  each  run. 

I  left  the  Simulation  Center  with  questions  burning  in  my  head.  As  a  tank 
officer,  I  had  used  simulation  all  of  my  career,  particularly  for  tank  gunnery.  I  had 
used  the  Army’s  Close  Combat  Tactical  Trainer  (CCTT),  where  huge  tank  mock- 
ups  complete  with  all  four  crew  positions  are  networked  together  to  fight  on  a 
virtual  battlefield.  However,  those  simulations  looked  and  sounded  like  tanks, 
albeit  with  many  deviations  from  the  real  thing.  Our  convoy  exercise  at  the 
Simulation  Center  was  different.  The  Marines  did  not  see  anything  that  day  that 
looked  remotely  like  a  High  Mobility  Multi-Wheeled  Vehicle  (HMMWV),  a 
machine  gun,  or  a  rifle.  They  saw  laptop  computers,  and  that  was  it.  I  had  seen 
personal  computers  used  for  simulation  before,  but  it  was  always  through  the 
very  abstract  level  of  operational  or  strategic  warfighting,  such  as  TACOPS, 
where  map  icons  represent  all  actions.  In  the  Simulation  Center  that  fall  day,  we 
were  not  having  blue  rectangles  meet  red  diamonds  and  wait  for  math  to  crunch 
through  the  computer  to  see  what  was  left.  The  Marines  were  fighting  as  their 
individual  selves,  quite  realistically  represented  in  the  virtual  world.  They  had 
gone  through  two  relatively  elaborate  exercises  in  less  than  six  hours  with  no 
logistics  footprint,  no  equipment,  and  no  permanently  established  bulky 
simulator. 

I  mulled  the  experience  over  for  the  next  few  weeks,  but  I  considered  it 
much  more  thoroughly  after  a  discussion  with  one  of  the  Marine  Corps’  few 

3 


Simulation  Officers.  He  explained  that  the  Marine  Corps  would  field  VBS  to  all 
infantry  battalions  within  a  couple  years.  Soon,  the  VBS  exercise  I  had  executed 
with  my  Marines  could  be  the  norm  for  one-third  of  the  Marine  Corps.  I  looked 
back  on  my  experience  with  alarm,  recognizing  that  I  had  known  very  little  about 
the  simulation  throughout  the  whole  ordeal.  I  was  not  even  certain  the  Marines 
had  gained  that  much  from  the  simulation.  They  benefitted  from  the  classes  a  bit 
and  certainly  the  planning  forced  them  to  get  into  a  combat  mindset  for  a  moment 
in  time.  Cverall,  the  program  was  better  than  nothing;  at  least  they  had  been 
forced  to  put  their  minds  back  in  a  combat  scenario.  Had  the  VBS  simulation 
contributed  to  making  the  Marines  tactically  better  at  their  jobs?  Cr,  worse,  had 
the  simulation  taught  them  things  that  were  not  right  in  the  real  world?  Certainly, 
we  had  experienced  “game-isms,”  peculiarities  associated  with  the  simulation 
exercise.  Did  VBS  really  make  them  better?  I  was  skeptical. 

Cnee  I  reported  to  the  Naval  Postgraduate  School,  I  started  to  discuss  this 
idea  with  faculty  and  was  surprised  at  how  little  knowledge  existed  about  VBS. 
Most  people  only  knew  about  the  simulation  from  a  cursory  back-of-the-brochure 
sort  of  viewpoint,  and  no  one  had  really  looked  into  how  effective  the  simulation 
was  or  what  skill  sets  it  best  suited.  The  Marine  Corps  had  already  spent  a  lot  of 
money  on  the  system  and  had  scheduled  its  distribution  to  the  fleet.  I  became 
fascinated  by  what  was  known,  and  equally  by  what  was  not  known,  about  the 
system.  Finally,  my  thoughts  and  experience  with  VBS  boiled  down  to  a  single, 
simple,  burning  question:  does  VBS  make  Marines  better  at  their  jobs?  I 
decided  to  devote  my  two  years  of  modeling  and  simulation  study  to  this 
question. 

B.  SCOPE 

I  originally  administered  a  platoon-level  convoy  exercise  because  it  fit  the 
needs  of  my  Marines,  but  the  proliferation  of  lEDs  throughout  CEF  and  CIF  has 
brought  this  scenario  into  the  public  and  military  consciousness  as  never  before. 
Figuring  out  how  to  tactically  drive  around  has  become  one  of  the  cornerstone 
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issues  of  Post-9-1 1  military  tactical  thought.  Moreover,  this  exercise  focuses  on 
the  senior  NCO  or  junior  Staff  NCO  on  whom  so  many  important  war  decisions 
rest.  Small  unit  tactical  convoy  is  certainly  an  exercise  that  trains  Marines  in  a 
skill  set  with  both  operational  and  tactical  relevance. 

We  chose  the  platoon-level  convoy  because  of  its  deep  relevance  in 
current  operations.  Convoy  operations,  once  considered  a  rear  echelon  low- 
intensity  combat  function,  have  come  to  the  forefront  of  military  thinking  in  OEF 
and  OIF.  Many  of  the  war’s  casualties  have  occurred  in  this  area.  Tactical 
convoys  are  often  planned  and  executed  at  the  platoon  level,  and  platoon 
leadership  can  be  required  to  make  life  and  death  decisions  in  tough  situations. 
The  operational  forces  have  quickly  ramped  up  convoy  training  since  the 
beginning  of  OEF,  and  this  aspect  of  simulation  training  deserves  immediate 
critical  scrutiny  to  provide  the  best  answers  for  the  Marines  and  soldiers  on  the 
ground. 

The  primary  purpose  of  this  thesis  is  to  explore  the  effectiveness  of  VBS 
2™  as  a  training  tool  for  small  unit  tactics.  Additionally,  the  thesis  seeks  to 
investigate  whether  the  simulation  itself  contributes  to  increased  training  or  if  the 
trainer,  influenced  by  training  and  experience,  affects  the  training  impact  of  the 
simulation.  Information  gained  from  exploration  of  the  second  question  is 
intended  to  suggest  best  implementation  practices  for  the  simulation,  assuming 
that  it  is  shown  to  be  effective.  However,  the  thesis  does  not  intend  to  generalize 
findings  broadly,  but  rather  focuses  on  platoon-level  tactical  convoy  training. 

C.  RESEARCH  QUESTIONS 

This  thesis  focuses  on  the  following  questions: 

•  Does  preliminary  training  with  VBS  2™  positively  impact 
performance  in  a  squad  level  tactical  scenario? 

•  Does  the  level  of  training  of  the  simulation  controller  impact  the 
effectiveness  of  VBS  2™  preliminary  training? 
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D.  METHODOLOGY 

1.  Background  Study 

We  started  the  project  by  researching  how  other  people  had  tackled  the 
problem  of  determining  whether  personal  computer  games  work  for  training.  We 
conducted  a  literature  review  focusing  on  two  major  topic  areas;  the  historical 
use  of  games  for  training  and  training  transfer  of  simulation  based  training.  This 
background  research  served  as  a  starting  point  for  the  project’s  work. 

2.  Pilot  Studies 

We  started  this  project  with  no  technical  knowledge  of  VBS  2™.  We  set 
up  some  preliminary  studies  to  learn  how  to  work  the  software  and  develop  a 
basis  of  experience  for  intelligent  use  of  the  simulation  as  a  training  tool.  We 
started  by  learning  how  to  operate  the  simulation  as  a  user  and  how  to  teach 
others  to  do  the  same.  We  then  learned  how  to  develop  scenarios  in  VBS  2™ 
and  tested  whether  they  were  working  as  designed.  To  see  whether  they 
worked,  we  first  tested  whether  the  scenarios  technically  performed  as  intended. 
Then  we  conducted  testing  to  determine  whether  the  scenarios  met  their  tactical 
objectives. 

3.  Experiment  in  Support  of  Simulation  Effectiveness  Question 

Once  we  had  developed  a  sound  base  of  experience  to  use  VBS  2™  for 
training,  we  developed  a  set  of  scenarios  to  support  training.  We  administered 
the  training  to  an  operational  unit  and  used  several  methods  to  determine  the 
effectiveness  of  the  training.  Methods  of  evaluation  included  surveys, 
performance  on  academic  knowledge  tests,  and  performance  in  live  training. 
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4.  Experiment  in  Support  of  Trainer  Effect  on  Effectiveness 
Question 

We  developed  a  training  program  to  train  the  trainer  on  the  use  of  VBS 
2™  to  accomplish  unit  goals.  We  administered  the  training  program  and  used 
survey  and  academic  knowledge  test  evaluations  to  determine  whether  the 
trainer  made  a  difference  to  the  effectiveness  of  the  simulation  training. 

5.  Conciusion 

We  concluded  the  study  with  a  series  of  observations  and 
recommendations.  We  also  described  some  future  research  that  could  be 
developed  from  this  project. 

E.  CHAPTER  OUTLINE 
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A.  Motivation 
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II.  BACKGROUND 


A.  INTRODUCTION 

The  U.S.  Marine  Corps  encourages  extracurricular  reading  through  the 
“Marine  Corps  Professional  Reading  Program.”  This  list  of  military  related  books 
divides  its  recommended  titles  by  rank  so  that  each  Marine  reads  material 
correctly  related  to  the  level  of  warfare  typical  of  his  or  her  pay  grade.  For  the 
most  part,  these  titles  take  Marines  back  into  the  history  of  warfare  or  military 
heritage.  They  may  provide  doctrinal  insights  or  relate  different  views  of  tactics, 
techniques,  and  procedures.  One  book  recommended  for  entry-level  Marines  at 
both  the  officer  and  enlisted  levels  stands  out  because  it  simply  does  not  fit  with 
these  other  works.  The  book  is  a  science  fiction  novel  from  the  mid-1980s  called 
Ender’s  Game  (U.S.  Marine  Corps,  n.d.). 

Orson  Scott  Card  published  Ender’s  Game  in  1985,  when  computers  were 
not  quite  established  in  every  household  and  the  Cold  War  still  defined  the 
American  understanding  of  what  war  was  and  what  a  military  did.  The  novel’s 
protagonist,  Ender,  was  a  12-year-old  boy  who  played  video  games  well.  He  did 
so  well,  in  fact,  that  the  government  sent  him  to  Battle  School  for  advanced 
training.  While  at  Battle  School,  he  learned  all  the  known  techniques  of  playing 
the  school’s  games  and  became  the  best  at  them  all.  The  government 
developed  Battle  School  to  immerse  students  in  the  gaming  environment  in 
preparation  for  armed  conflict  with  the  buggers,  an  insect-like  alien  race  that 
threatened  to  overwhelm  Earth.  Ender  excelled  in  this  environment,  taking  on 
increasingly  more  difficult  simulations.  Finally,  Ender  fought  in  a  culminating, 
enormously  challenging  computer  game.  Feeling  his  options  close  in  after  a 
particularly  overwhelming  series  of  actions,  Ender  made  a  risky  move  in  the 
game  and  consequently  won  the  day.  Exhausted,  he  left  the  simulator  to  find  out 
the  secret  of  the  game  and  of  Battle  School  itself.  His  mentor  and  instructor 
explained: 
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Ender,  for  the  past  few  months  you  have  been  the  battle 
commander  of  our  fleets.  This  was  the  Third  Invasion.  There  were 
no  games,  the  battles  were  real,  and  the  only  enemy  you  fought 
was  the  buggers,  (p.  296) 

Ender,  in  fact,  had  been  remotely  controlling  a  fleet  of  ships  fighting  the  real 
buggers.  Some  of  his  old  friends  from  Battle  School  had  been  piloting  the  ships 
under  his  command  (Card,  1985). 

This  thesis  is  about  simulation,  so  the  idea  of  including  a  book  in  the 
Professional  Reading  Program  whose  theme  espouses  such  a  triumph  of 
simulation  is  most  intriguing.  However,  Ender’s  Game  offers  more  as  a  reading 
list  recommendation  than  this.  The  Marine  Corps  most  likely  included  this  book 
as  an  illustration  of  the  close,  and  sometimes  indistinguishable,  relationship 
between  training  and  war.  The  idea  of  Ender  unwittingly  controlling  a  living, 
breathing,  and  dying  army  while  playing  a  video  game  is  novel  and  thought- 
provoking.  He  was  training  for  war.  Ultimately,  every  commander  yearns  for  this 
level  of  training  in  any  given  scenario.  That  is,  the  training  should  be  so  real  that 
one  cannot  tell  the  difference  from  actual  combat.  The  military  should  seek  this 
level  of  training  in  all  endeavors,  so  that  servicemembers  execute  in  exactly  the 
same  way  during  peace  or  war  with  the  state  of  hostilities  being  irrelevant. 

Michael  Macedonia,  the  technology  officer  for  Program  Executive  Office 
Simulation,  Training,  and  Instrumentation  Command  (PEO  STRI),  looked  at  the 
inseparability  between  training  and  the  reality  of  war  another  way.  He  hailed  a 
scene  from  the  movie  Patton  as  illustrative  of  his  point.  In  the  scene.  General 
Patton  approached  an  open  area  while  directing  his  driver  to  a  battle  scene.  He 
described  to  his  driver  a  scene  of  ancient  battle  between  the  Carthaginians  and 
the  Romans  far  removed  from  the  movie’s  World  War  II  engagements.  Then, 
Patton  said,  “I  was  here.”  Macedonia  pointed  out  that  the  scene  painted  a 
picture  of  a  man  so  engrossed  in  the  study  of  war  history  that  events  of  his 
studies  became  their  own  reality  in  the  general’s  mind.  General  Patton’s  literary 
education  created  memories  in  his  mind  that  were  not  there  before.  Macedonia 
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explained  that  simulation  boils  down  to  the  ultimate  objective  of  creating 
memories.  Soldiers  and  Marines  on  the  battlefield  against  a  fighting  enemy 
should  look  at  their  situation  and  recognize  that  they  were  there,  that  they  have 
seen  this  before  (Halter,  2006).  In  this  way,  Macedonia  sets  the  bar  for 
simulation,  games,  and  training.  An  individual  who  has  used  the  training  tool 
should  approach  the  real  event  knowing  its  subtleties  and  nuances  from  thorough 
exploration  with  the  tool.  Like  Ender,  the  soldier  or  Marine  should  participate  at 
such  a  level  that  the  fact  of  a  real  adversary  becomes  immaterial;  the  skills  and 
their  application  are  the  same. 

This  thesis  invites  exploration  into  two  general  domains.  First,  military 
games  have  a  rich  history  and  that  history  must  hold  some  nuggets  of  truth  about 
what  games  can  and  cannot  do  for  the  warfighter.  Second,  concern  over  the 
effectiveness  of  simulation  is  not  a  new  topic.  Studies  wrestling  with  such 
questions  are  typically  lumped  under  the  umbrella  of  “training  transfer.”  A  review 
of  past  training  transfer  efforts  should  shed  some  light  on  the  Best  Practices  for 
evaluating  simulation  training  systems. 

B.  EVOLUTION  OF  PERSONAL  COMPUTER-BASED  TRAINING  GAMES 

Humans  have  a  history  of  playing  games  of  the  mind  and  of  the  body  that 
seems  as  old  and  as  diverse  as  any  other  activity.  Today,  an  array  of  bright, 
shiny,  new  objects  stands  before  the  services  as  the  cutting  edge,  innovative 
training  of  the  future.  Personal  computer-based  simulations  are  these  objects, 
and  they  have  gained  attention,  both  skeptical  and  enthusiastic.  Some 
researchers  have  called  this  genre  of  training  device  “serious  games,”  to  thwart 
the  idea  that  our  troops  are  going  to  the  arcade  to  learn  to  fight.  One  might 
question  just  how  new  and  innovative  this  idea  really  is. 

1.  Games  Have  Been  Around  a  Long  Time  -  Just  Like  War 

The  story  has  it  that  the  father  of  a  young  Indian  prince  died  before  he 
could  teach  his  son  the  skills,  tactics,  and  strategies  of  war.  The  prince’s 
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advisors  fretted  over  how  to  prepare  the  prince  for  battle  with  known  adversaries 
that  imminently  threatened  the  kingdom.  At  a  loss  for  a  better  training 
mechanism,  the  advisors  introduced  the  young  prince  to  chess,  a  game  that  he 
played  regularly.  Later,  after  a  victorious  rout  on  the  field  of  battle,  the  prince 
attributed  his  success  to  the  development  of  his  mind  with  the  chessboard 
(Forbes,  1860).  The  story  is  a  mere  anecdote,  certainly  of  questionable  value  as 
historic  fact.  However,  the  idea  that  someone  thought  up  the  story  illustrates  the 
long-standing  close  link  between  games  and  war.  The  origins  of  chess  provide 
grounds  for  debate,  but  the  game  clearly  has  roots  in  an  Indian  game  called 
Chaturanga.  In  that  game,  the  pieces  were  modeled  after  the  battlefield  assets 
of  the  day,  such  as  elephant  and  horse,  and  two  players  took  turns  moving  about 
a  gridded  square  according  to  movement  rules  similar  to  chess  (Caffrey,  2000). 
One  simply  cannot  ignore  these  martial  roots. 

A  strategy  game  called  Wei  Ch’l,  or,  in  popular  western  parlance.  Go,  is 
hailed  as  the  oldest  strategy  game  still  played  in  its  original  state.  Archaeological 
evidence  points  to  ancient  Chinese  origins.  The  game  of  Go  spread  across  the 
entire  Orient  and  is  still  played.  Legend  suggests  that  the  game  has  military 
origins.  Encirclement  is  a  fundamental  feature  of  the  game,  and  the  ancient 
Chinese  used  this  strategy  of  dominance  both  in  hunting  and  warring  tactics.  Go 
is  notable  for  many  reasons.  The  game  was  extremely  widespread  and  enjoyed 
a  long  history  that  still  lives  on.  Go  has  been  recognized  as  a  domain  of  the  elite, 
and  proficiency  in  Go  has  been  a  criterion  for  the  elite  in  China,  Japan,  and  other 
Asian  countries.  In  fact,  Japan  formalized  Go  as  a  fundamental  part  of  the 
Shogun’s  court  in  the  medieval  period.  Most  importantly.  Go  influenced  many 
aspects  of  life  in  Asia  including  religion,  philosophy,  politics,  and  war.  Go  is  often 
linked  with  the  yin  and  yang  of  Taoism.  Go  influenced  such  thinkers  as  Sun  Tzu, 
the  author  The  Art  of  War,  the  oldest  military  doctrine,  which  is  a  book  that 
serves  players  of  Go  as  well  as  warfighters.  Mao  Tse-tung  reputedly  used  Go  in 
the  development  of  his  military  and  political  theories.  Go’s  importance  stems 
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from  its  widespread  impact  on  a  whole  society  throughout  history,  not  only  as  a 
pastime,  but  as  a  way  of  thinking  about  life  and  applying  those  ideas  practically 
(Shotwell,  2003). 

These  games  from  ancient  history  tell  us  very  little  about  how  to  use  a 
game  to  train  a  military  fighting  man  today.  The  depth  of  the  history,  however, 
indicates  that  games  have  been  used  to  develop  and  influence  military  strategy 
far  back  into  the  beginnings  of  war  itself.  The  idea  of  abstracting  the  concepts  of 
war  into  a  game,  with  systems  of  rules  and  randomization  that  individuals  can 
play  out  against  other  minds,  is  not  new  in  any  way.  Techniques  have  matured 
over  the  years,  as  the  next  sections  will  show,  but  the  theme  of  gaming  for 
military  benefit  has  deep  roots.  One  can  trace  the  games  just  discussed  back  to 
the  fifth  century  BC.  The  serious  game  is  a  new  term,  but  it  is  not  a  new  idea. 

2.  Games  Do  Not  Have  to  Be  Limited  to  Strategic-Levei 
Abstractions 

Gaming  for  training  began  to  evolve  beyond  the  confines  of  the 
chessboard  in  the  mid-1600s.  In  England,  the  King’s  Game  was  used  as  a 
training  device.  With  thirty  pieces  to  a  side  and  a  more  robust  rule  set  than 
chess,  the  game  added  realism  beyond  chess.  Meanwhile,  the  French  used  card 
games  to  train  military  professionals.  These  games  featured  actual  engagement 
situations,  thereby  extending  realism  beyond  the  chessboard.  The  Prussians 
also  used  games  for  training,  relying  on  the  math  and  geometry  of  the  game  to 
replicate  war  as  a  science  (Brewer  &  Shubik,  1979).  In  fact,  it  was  a  Prussian 
named  Baron  von  Reisswitz  who  is  credited  with  fathering  modern  wargaming. 

The  Germans  coined  the  term  kriegsspiel,  translated  “wargame”  in 
English.  No  thesis  written  by  a  Marine  would  be  complete  without  a  tribute  to  a 
dead  German,  and  this  one  will  be  no  different.  Baron  von  Reisswitz  served  the 
state  of  Prussia  as  the  war  counselor  at  Breslau.  In  1811,  he  introduced  a 
training  concept  that  has  changed  very  little  over  time,  considering  the 
technological  breakthroughs  of  the  two  centuries  since  then.  Officials  moderated 
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a  game  played  on  a  table  with  actual  terrain  modeled  on  top.  Units  represented 
by  blocks  moved  across  the  table  in  response  to  orders  from  the  players.  The 
moderating  officials  received  the  orders  from  the  players  and  updated  the  table 
accordingly.  In  order  to  adjudicate  battlefield  effects,  they  used  tables  to 
calculate  attrition  based  on  factors  such  as  range  and  terrain.  The  game  was  not 
deterministic  though,  because  the  final  outcome  faced  the  uncertainty  of  a  dice 
throw.  Many  of  the  advanced  and  highly  used  simulations  of  today  owe  their 
conceptual  design  to  this  early  innovation,  with  the  only  significant  difference 
being  the  added  computational  power  of  modern  computers.  Nevertheless,  for 
all  of  the  evidence  of  Reisswitz’s  contributions  in  today’s  war  simulations, 
Reisswitz  did  not  revolutionize  warfare,  battlefield  planning,  or  combat  training  in 
1811.  Kriegsspiel  merely  served  as  an  amusement  for  the  elite,  never  to  be  used 
by  the  fighting  forces  on  the  ground.  Baron  von  Reisswitz’s  son  brought 
kriegsspiel  to  the  common  fighting  man  when  he  altered  the  game  so  that  players 
could  use  a  map,  making  the  game  much  more  portable.  In  1824,  the  Prussian 
chief  of  staff  General  Karl  von  Muffling  liked  the  idea  so  much  that  he  ordered  all 
garrisons  to  use  it  (Caffrey,  2000). 

In  1837,  General  von  Moltke  took  the  ideas  of  Baron  von  Reisswitz  and 
his  son  and  turned  them  into  a  bona  fide  training  regime.  He  incorporated 
wargaming  with  staff  rides  to  train  the  officer  corps  of  the  Prussian  army.  As  the 
Prussians  demonstrated  prowess  on  the  early  nineteenth  century  battlefield,  the 
other  nations  of  the  world  took  notice  and  started  using  wargames  of  their  own. 
Spenser  Wilkinson,  a  British  college  student  vacationing  in  Germany  in  1873, 
exemplified  this  world  attention.  After  noticing  Britain’s  numerical  inferiority  to  the 
rest  of  Europe  in  a  pamphlet,  he  returned  to  England  and  organized  the  country’s 
first  wargaming  club.  In  the  United  States,  Major  Livermore  started  copying 
German  wargames  in  1883.  He  improved  the  Germans’  attrition  tables  with 
updated  statistics  from  the  American  Civil  War.  The  U.S.  Army’s  Chief  of  Staff, 
General  William  T.  Sherman,  disapproved  of  the  wargame  because,  in  the  game, 
units  fought  to  the  last  man,  something  experience  had  proven  unrealistic.  About 
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the  same  time,  William  McCarty  Little  fathered  naval  wargaming  in  America.  A 
key  player  in  the  formation  of  the  Naval  War  College,  McCarty  Little  instituted 
wargaming  there  in  1889,  where  wargames  have  occurred  every  year  since.  The 
Army  followed  this  practice  in  its  own  war  college  starting  in  1899  (Caffrey, 
2000). 

Wargaming  demonstrated  both  its  power  and  its  shortcomings  in  the 
preparations  for  World  War  I.  Both  the  Germans  and  the  British  analyzed  Count 
Alfred  von  Schlieffen’s  invasion  plans,  and  both  governments  used  wargame 
results  to  make  national  decisions  about  military  preparation.  However,  the 
German  wargaming,  meticulous  and  calculated  as  it  was,  failed  to  anticipate 
diplomatic  and  political  consequences  that  were  seen  in  their  defeat.  While  most 
of  the  nations  who  participated  in  World  War  II  used  wargaming  in  the  inter-war 
years,  the  Navy  and  Marine  Corps  had  the  champion  effort.  The  Navy  developed 
the  island-hopping  theory  that  would  eventually  become  the  backbone  of  the 
Pacific  campaign,  while  the  Marine  Corps  changed  the  face  of  amphibious 
doctrine  through  wargames  in  lieu  of  a  budget  for  live  amphibious  training.  By 
far,  the  most  significant  wargamed  event  of  this  time  was  the  Japanese  attack  on 
Pearl  Harbor,  where  wargaming  radically  changed  the  original  plan,  producing 
the  intended  results.  All  major  players  in  World  War  II  used  wargaming  to  some 
extent,  such  that  wargaming  was  an  accepted  military  practice  by  the  end  of  the 
conflict  (Caffrey,  2000). 

Reisswitz  brought  us  the  red  and  blue  of  war.  He  color-coded  his  original 
nineteenth  century  tokens  this  way,  setting  a  standard  for  military  lingo  today. 
More  than  just  colors,  Reisswitz  introduced  a  way  of  visualizing  the  battlefield 
and  its  infinite  layers  of  possibilities.  Unlike  the  strategic  games  of  Go,  chess, 
and  patteia  that  were  played  at  the  strategic  level  in  the  homes  of  the  elite, 
Reisswitz’s  style  of  wargaming  influenced  the  warfighter  himself  at  the 
operational  level  of  war.  As  military  actions  such  as  Pearl  Harbor  illustrated. 
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wargaming  had  much  more  power  than  providing  a  theoretical  forum  for  abstract 
concepts:  wargaming  started  to  change  war  itself,  as  militaries  instituted  it  as  a 
standard  battle  preparation  activity. 

3.  Wargaming  Links  the  Living  Room  and  the  Command  Post 

“A  gun  capable  of  hitting  a  toy  soldier  nine  times  out  of  ten  at  a  distance  of 
nine  yards”  sounds  like  the  perfect  Christmas  gift  for  a  young  boy.  This  quote 
comes  from  the  exuberant  text  of  H.  G.  Wells’  1913  wargaming  manual  Little 
wars.  Wells  was  not  excited  about  this  toy  gun  for  his  children;  rather,  he  was 
using  the  toy  along  with  paper  houses,  miniature  trees,  and  toy  soldiers  to  build 
his  own  battlefields  on  his  lawn  or  living  room  floor.  The  roots  of  Wells’  work 
started  with  “lunching  with  a  friend”  and  evolved  into  a  sophisticated  explanation 
of  how  to  model  war  in  one’s  home,  complete  with  a  full  example  battle  (Wells, 
1970).  Robert  Louis  Stevenson  is  another  notable  author  who  engaged  in  war 
games,  devoting  much  of  a  three-year  convalescent  period  to  the  development  of 
a  detailed  war.  Sir  Winston  Churchill’s  childhood  war  games  with  his  brother. 
Jack,  influenced  his  decision  to  join  the  military.  Churchill  never  lost  his  love  of 
playing  with  toy  soldiers,  taking  the  hobby  into  his  adult  years  (Featherstone, 
1962). 

Toy  soldiers  emerged  in  European  culture  on  the  heels  of  Reisswitz’s 
wargame.  Daddy  played  with  lead  figurines  to  re-enact  history  or  to  learn  the 
best  tactics,  and  Junior  wanted  a  piece  of  the  action.  War  made  great  toys,  and 
World  War  I  sparked  its  own  set  of  miniature  replicas  of  machines  of  destruction. 
In  fact,  toy  soldiers  grew  into  an  industry  to  satisfy  both  youth  and  adults.  This 
play  has  never  gone  away.  From  plastic  “army  men”  to  the  Gl  Joe  action  figures, 
youth  have  delighted  in  toys  that  depict  war.  H.  G.  Wells  captures  it  all  in  his 
subtitle  to  Little  wars:  “A  game  for  boys  from  twelve  years  of  age  to  one  hundred 
and  fifty  and  for  that  more  intelligent  sort  of  girl  who  likes  boys’  games  and 
books.”  Many  people  want  to  be  the  general,  if  only  for  a  moment,  whether  it  be 
the  boy  in  his  room,  the  adult  hobbyist  at  his  couch,  or  the  actual  general  himself 
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making  plans  to  defeat  the  enemy  (Featherstone,  1962).  This  fundamental 
inseparability  between  the  living  room  and  the  war  room  contributes  to 
understanding  the  strengths  and  weaknesses  of  the  twenty-first  century  serious 
games. 

4.  A  Soldier  Can  Interface  With  an  Electronic  Device  to  Train 

In  the  1930s,  Edwin  Link  did  not  have  money  for  a  plane  to  learn  to  fly,  so 
he  developed  a  plan  to  help  people  learn  to  fly  without  an  actual  aircraft.  He  built 
a  simulator.  His  wooden  box  mounted  on  a  crude  motion  platform  did  not  look 
much  like  its  modern  successors  of  today,  but  it  provided  enough  realism  for 
people  to  learn  the  basics  of  aviation,  particularly  in  the  area  of  instrument  flight. 
Link’s  work  might  have  gone  largely  unnoticed  if  an  unfortunate  series  of  Army 
Air  Corps  accidents  had  not  attracted  attention  to  flight  training.  In  1934,  Link 
amazed  Army  officers  by  safely  landing  his  plane  out  of  a  cloudy  sky  by  flying 
completely  on  instruments.  The  Army  Air  Corps  contracted  with  Link  to  build 
simulators  for  the  military  so  that  pilots  could  learn  instrument  flying.  Link’s 
trainer  contained  a  mock  instrument  panel  and  the  basic  plane  controls.  With 
these  tools,  students  could  learn  to  fly  without  looking  at  the  ground.  Link’s 
company  produced  over  10,000  of  his  “Blue  Boxes”  during  World  War  II,  and 
simulation  became  a  staple  of  military  pilot  training  (L3  Communications,  2010). 

Link’s  trainer  has  very  little  to  do  with  gaming  itself,  but  it  is  relevant  to  this 
study  because  of  its  introduction  of  the  virtual  world.  Until  the  twentieth  century, 
training  at  the  soldier  level  involved  learning  basic  skill  sets,  such  as  operation  of 
a  firearm  or  short  scripted  movements  in  the  form  of  battle  drills.  World  War  I 
changed  all  that  with  complicated  machinery,  like  the  tank,  machine  gun,  and 
airplane;  equipment  that  could  influence  the  battlefield  like  never  before. 
Suddenly,  an  individual  soldier’s  relationship  to  a  particular  machine  mattered. 
The  idea  that  the  military  needed  to  train  this  relationship  at  all  was  novel,  and 
Link’s  idea  of  doing  so  through  simulation  was  truly  ahead  of  his  time.  Though 
crude.  Link’s  trainer,  through  its  instruments  and  motions,  provided  the  trainee  a 
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visualization  of  a  world  that  did  not  exist.  Link  invented  the  virtual  world,  before 
the  computer  made  such  things  commonplace.  It  is  worth  noting  that  the  first 
virtual  world  was  developed  for  military  training. 

5.  The  Computer  Adds  New  Possibilities  for  Wargaming 

After  World  War  II,  concern  over  how  to  fight  a  nuclear  war  gave 
wargaming  a  new  boost,  since  this  was  the  only  venue  available  to  analyze 
nuclear  tactics  and  strategy.  During  the  1950s,  the  Air  Force  led  the  Department 
of  Defense  to  work  with  RAND  Corporation  to  develop  wargaming  as  a  complex 
mathematical  exercise.  RAND  created  some  ideas  that  stuck,  such  as  the 
gridded  playing  board,  tables  to  calculate  the  results  of  combat  actions 
mathematically,  randomization  through  dice,  rules  for  playing  by  turns,  and  the 
incorporation  of  terrain.  Charles  Roberts  conducted  a  parallel  game 
development  effort  that  led  to  the  creation  of  the  commercial  entertainment 
company  Avalon  Hill,  in  1958.  RAND  brought  mathematical  gaming  to  the 
military,  and  Avalon  Hill  took  it  to  the  living  room  (Smith,  2008).  Through  the 
1950s  and  1960s,  this  mathematically  based  style  of  military  wargaming 
matured.  Most  notably,  though,  wargaming  remained  mainly  at  the  strategic  or 
operational  levels,  with  games  oriented  on  winning  by  destroying  more  personnel 
and  materiel  than  the  other  players  destroyed.  There  was  some  recognition  that 
wargaming  needed  to  extend  beyond  attrition.  For  example,  a  1964  Advanced 
Research  Projects  Agency  (ARPA)  project  sought  a  wargame  that  would 
demonstrate  political,  economic,  and  psychological  aspects  of  an  insurgency,  but 
the  research  efforts  did  not  go  very  far  in  changing  the  face  of  attrition  gaming 
(Caffrey,  2000).  Such  wargaming  efforts  continued  to  mature  throughout  the 
1970s,  in  both  military  and  civilian  sectors.  While  military  and  civilian  gaming 
remained  largely  separate  efforts,  it  would  be  a  mistake  to  discount  any 
relationship  between  the  two.  For  example,  in  1974,  the  Army  purchased  a 
commercial  wargame  called  Fire  Fight  to  supplement  its  wargaming  collection 
(Caffrey,  2000). 
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These  mathematical  models  of  war  increased  in  accuracy  as  the  math 
increased  in  complexity.  Eventually,  computers  broke  the  mold  for  wargaming. 
At  first,  computers  served  only  as  glorified  calculators,  but  even  this  advance 
allowed  wargaming  complexity  to  develop.  The  mechanics  of  game  play 
remained  the  same  with  playing  boards,  lookup  tables,  representational  pieces, 
and  dice,  but  computers  sped  up  the  calculations  allowing  more  precise 
numerical  modeling.  During  the  1970s  and  1980s,  the  wargame  moved  from  the 
playing  board  to  the  computer  itself  as  computers  grew  powerful  enough  for  the 
job.  Eventually,  computers  supported  gaming  well  enough  to  offer  advantages 
never  fathomed  in  the  days  of  the  playing  board.  Players  could  work  in  separate 
rooms  with  customized  views.  They  could  make  decisions  and  implement  them 
in  real  time  with  many  others  doing  the  same.  The  age  of  the  computer  game 
had  begun.  Because  computers  were  still  too  expensive  and  bulky  for  the 
average  consumer,  these  games  stayed  military.  The  games’  roots  in  strategic 
and  operational  board  gaming  kept  simulation  at  the  higher  levels  of  war. 

Wargaming  and  war  modeling  became  integral  components  of  training 
and  planning  during  the  1990s.  The  rise  of  the  personal  computer  made 
simulations  such  as  JANUS  and  Modular  Semi-Automated  Forces  (ModSAF) 
widespread.  Simulations  like  these  proved  themselves  during  the  planning  and 
execution  of  the  Gulf  War  (Smith,  2008).  All  services  used  such  games  in 
training  throughout  the  1990s,  and  many  joint  efforts  brought  the  services 
together  in  simulated  environments.  The  1991  creation  of  the  Defense  Modeling 
and  Simulation  Office  (DMSO)  demonstrated  the  importance  of  the  community, 
and  simulations  such  as  the  Joint  Simulation  System  (JSIMS)  and  the  Joint 
Warfare  System  (JWARS)  showed  efforts  to  bring  everyone  together  in  the  joint 
arena  (Caffrey,  2000). 
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6. 


The  Military  Can  Train  Cheaply  Through  Adaptation  of  Civilian 
Games 


In  1995,  General  Krulak,  the  U.S.  Marine  Corps  Commandant  at  the  time, 
issued  his  Planning  Guidance  that  provided  the  broad  vision  for  the  service  for 
the  next  four  years.  The  following  directive  ranked  third-highest  in  the  priority  list: 
“Make  our  education  and  training  processes  and  institutions  technologically 
innovative,  challenging,  and  fun”  (Jernigan,  1997).  General  Krulak  even  added 
teeth  to  this  task  with  the  creation  of  the  Marine  Warfighting  Lab  to  test  new 
strategies,  technologies,  and  ideas.  The  Commandant’s  Planning  Guidance 
opened  new  horizons  for  Quantico’s  Marine  Corps  Modeling  and  Simulation 
Office  (MCMSO).  A  couple  of  Marines  searched  all  available  civilian  gaming 
systems  to  find  anything  useful  for  military  training.  As  a  marketing  ploy,  id 
Software  had  released  parts  of  Doom  II  as  shareware  for  players  to  modify. 
Seizing  the  opportunity.  Sergeant  Daniel  Snyder  spent  three  months  of  his  off- 
duty  time  changing  Doom  II  into  Marine  Doom,  an  application  with  military 
avatars.  Ml 6  rifles,  and  realistic  survival  rates  (Jernigan,  1997).  As  many  as  four 
players  could  fight  each  other  in  a  two-dimensional  urban  environment  of  walls 
and  passageways.  For  the  price  of  a  $49.95  Doom  II  CD-ROM,  the  Marine 
Corps  had  built  a  simulation  capable  of  teaching  individual  and  fire  team  skills. 
Since  the  game  resided  on  a  CD-ROM,  Marines  could  use  it  on  any  computer, 
playing  the  game  at  their  desks  or  taking  it  on  deployment  wherever  the  Corps 
might  send  them  (Riddell,  1997). 

Marine  Doom  did  not  come  into  the  Corps  as  just  a  fly-by-night  affair. 
While  the  program  owed  its  existence  to  the  innovation  of  fewer  than  a  handful  of 
Marines,  the  project  reflected  imagination,  analysis,  and  forward  thinking.  Merely 
gaining  acceptance  of  any  such  game  was  a  Sisyphean  task.  Ironically, 
Sergeant  Snyder  had  installed  Doom  on  the  hard  drive  of  his  computer  a  few 
years  earlier  in  communications  school.  In  doing  so,  he  had  violated  a  Marine 
Corps  order  prohibiting  computer  games  on  military  hard  drives  and  received 
nonjudicial  punishment  for  his  actions.  In  1995,  General  Krulak’s  guidance 
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changed  everything,  and  Marine  Doom’s  prospects  looked  bright  with  the  full 
support  of  the  Commandant.  Sergeant  Snyder  and  his  Officer  in  Charge, 
Lieutenant  Scott  Barnett,  had  developed  a  training  system  that  met  Marine  Corps 
training  objectives  of  interlocking  fields  of  fire,  use  of  cover  and  concealment,  and 
effective  communications  (Riddell,  1997).  Marine  Doom  joined  a  small  list  of 
twelve  simulations  allowed  on  Marine  computers  (Jernigan,  1997).  For  the  next 
few  years.  Marines  fought  each  other  in  their  offices  during  lunch  breaks.  The 
program  slowly  slid  into  extinction  as  better  technologies  developed.  Game 
problems,  such  as  player  tunnel  vision,  unrepresented  firing  positions,  and 
weapons  that  never  overheated,  came  to  the  surface  (Jernigan,  1997).  Marine 
Doom  only  featured  a  two-dimensional  battle  space,  a  particularly  limiting 
problem  when  simulating  urban  combat.  Lurking  in  the  background  of  this  early 
experiment  was  the  haunting  question  of  how  much  good  such  technology  could 
do  and,  worse,  whether  it  could  do  harm  by  teaching  players  the  wrong 
techniques.  With  such  a  rags-to-riches  start,  eventually  gaining  the  enthusiastic 
support  of  the  Commandant  himself.  Marine  Doom  soared  to  great  heights  as  a 
training  platform,  but  faded  into  obscurity  almost  as  rapidly.  In  the  end, 
commanders  did  not  see  the  training  results.  Two  Marines  had  successfully 
leveraged  commercial  off-the-shelf  (COTS)  technology,  but  the  training  simply 
did  not  transfer  to  the  warfighter.  The  game  disappeared.  The  ground  had  been 
broken,  however;  civilian  computer  gaming  technology  could  be  levied  for  military 
use. 

General  Krulak’s  guidance,  coupled  with  Sergeant  Snyder’s  proactive 
execution,  had  demonstrated  the  potential  for  using  the  shelves  of  the  local 
computer  store  for  military  training,  but  Sergeant  Snyder  was  not  the  first  to  use 
this  technique.  In  1980,  Atari  introduced  the  first-person  shooter  game  with 
Battlezone.  This  early  video  game  allowed  a  player  to  act  as  a  tank  gunner  and 
engage  enemy  tanks  on  a  three-dimensional  battlefield.  The  player  looked 
through  a  small  viewfinder  window  and  manipulated  controls  like  those  in  a  real 
tank.  Users  could  prowl  around  a  virtual  world,  albeit  one  composed  of  the 
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sparse  graphics  of  the  time,  complete  with  three-dimensional  terrain.  Battlezone 
simplified  the  battlefield  into  sterile  tank  on  tank  battles  with  no  people,  just  tanks 
that  either  lived  or  died.  Nevertheless,  players  immersed  themselves  in  the 
soldier’s  battlefield  for  the  first  time,  fighting  from  the  arcade  just  like  a  soldier  in 
a  real  tank  (Halter,  2006). 

The  Army  recognized  the  potential  for  the  addictive  game  play  to  turn  a 
leisure  activity  into  valuable  training.  The  service  persuaded  Ed  Rotberg,  Atari’s 
Battlezone  designer,  to  transform  the  game  into  a  military  training  device.  After 
three  months  of  long  hours  and  hard  work,  Rotberg  produced  Army  Battlezone. 
General  Donn  A.  Starry  was  the  commanding  general  of  TRADOC  at  the  time, 
and  he  oversaw  Atari’s  project.  General  Starry  recognized  that  soldiers  had 
grown  up  in  an  environment  of  electronic  gadgetry,  and  teaching  methods  for  this 
new  breed  had  to  be  developed  accordingly.  In  a  1981  conference,  he  asked  a 
question  that  is  still  quite  relevant: 

In  an  era  that  has  seen  such  fantastic  technological  achievements, 
how  is  it  that  our  soldiers  are  still  sitting  in  classrooms,  still  listening 
to  lectures,  still  depending  on  books  and  other  paper  reading 
materials,  when  possibly  new  and  better  means  for  training  have 
been  available  for  many  years?  (Halter,  2006) 

General  Starry  had  introduced  the  training  concept  that  General  Krulak  worked 
so  hard  to  put  into  practice  fifteen  years  later.  The  products  in  the  teenager’s 
arcade  were  fair  game  for  military  training,  if  only  the  military  applied  the  right 
imagination  and  planning. 

7.  Games  Can  Benefit  the  Individual  Warfighter 

While  the  wargaming  discussed  thus  far  impacted  strategy  for  centuries, 

this  type  of  gaming  does  nothing  for  the  individual  warfighter.  The  soldier  or 

Marine  on  the  ground  needs  something  at  the  tactical  level.  Before  computers, 

the  Tactical  Decision  Exercise  (TDE)  accomplished  this  objective.  A  TDE  poses 

a  static  military  situation  to  the  participant  in  the  form  of  a  simple  sketch  or 

composite  of  map  graphics.  The  participant  individually  develops  a  solution  for 
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the  situation,  usually  in  a  limited  amount  of  time,  and  prepares  the  orders  that  he 
or  she  would  give  subordinates  to  execute  his  or  her  plan.  IDEs  orient  on  an 
individual’s  decision-making  ability.  Rarely  is  the  plan  itself  the  point  of  the 
exercise;  most  often,  the  rationale  that  led  to  the  plan  produces  the  real  learning. 
IDEs  offer  teaching  power  by  pitting  the  mind  of  the  participant  against  the 
experience  of  a  mentor  or  instructor.  Certainly,  the  TDE  provides  a  forum  to 
discuss  tactics,  techniques,  and  procedures  in  a  practical  application.  More  than 
that,  IDEs  provide  a  window  into  the  mentor’s  mind,  allowing  the  student  to  see 
how  to  work  through  a  problem  or  to  see  how  the  mentor  thinks.  In  this  way, 
IDEs  provide  a  mechanism  to  convey  commander’s  intent  and  battlefield  vision. 
Like  the  board  games  previously  discussed,  computers  take  IDEs  to  new  levels. 
Virtual  worlds  allow  participants  to  see  their  plans  through  a  simulated  execution. 
As  Major  Brewster  puts  it  in  his  2002  article  on  the  subject,  “Simulation  allows  the 
student  to  progress  to  the  point  where  he  can  observe  the  ramifications  of  his 
decisions”  (Brewster,  2002).  An  understanding  of  IDEs  is  crucial  to  harnessing 
the  power  of  personal  computer-based  games. 

The  Marine  Corps  uses  a  simulation  called  Close  Combat  Marine  (CCM) 
to  stimulate  decision  making  at  the  junior  leadership  level.  The  simulation 
provides  a  simplistic  two-dimensional  map  interface  for  multiple  players  to 
manipulate  units,  usually  at  the  squad  level.  Using  the  map,  a  trainer  can 
develop  a  scenario,  and  Marines  can  maneuver  forces  and  watch  the  results  of 
their  decisions.  Quite  simply,  CCM  is  a  natural  computer  extension  of  the  TDE. 
In  2007,  two  officers  at  the  Naval  Postgraduate  School  conducted  a  study  to 
determine  whether  CCM  introduced  any  training  advantage  over  the  more 
traditional  TDE.  They  found  that  both  training  media  provided  benefit,  but  that 
only  the  computer  simulation  provided  a  means  to  evaluate  situational 
awareness  (Fitzpatrick  III  &  Ayvaz,  2007).  Because  of  the  importance  of 
situational  awareness  in  team  leadership  and  decision  making,  this  finding 
indicated  that  computer  games  offered  the  individual  soldier  or  Marine  something 
that  could  not  be  obtained  with  just  paper,  pencil,  and  tokens  on  a  table. 
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A  simplified  version  of  kriegsspiel  has  provided  a  training  medium  for  the 
individual  soldier  or  Marine  for  years.  Sand  table  exercises  involve  the  use  of 
small  replicas  of  battlefield  forces  on  a  model  of  the  terrain  to  demonstrate, 
analyze,  or  practice  tactical  maneuver  and  fires.  While  this  sort  of  gaming  has  a 
rich  history  and  can  become  quite  complex,  implementations  at  the  small  unit 
level  are  usually  quite  simple.  The  terrain  model  might  be  a  ruffled-up  bed  cover 
in  a  barracks  room,  a  dirt  patch  in  a  common  area  outside,  or  an  actual  table  top 
with  sides  holding  a  sand  box  that  easily  models  hills  and  valleys.  Force  replicas 
might  be  simple  toy  military  men  and  vehicles,  miniature  vehicle  replicas,  or 
rocks  available  on  the  ground.  Typically,  trainees  practice  a  scenario  by 
timeframe,  with  each  key  player  describing  actions  he  would  take  at  prescribed 
times  and  locations.  A  controller  can  then  assess  the  effects  of  the  actions  to 
determine  how  the  succeeding  timeframe  will  begin.  In  a  study  involving  a 
comparison  of  the  sand  table  exercise  and  other  traditional  training  with 
computer-based  games.  Majors  Nolan  and  Jones  noted  that  some  participants 
preferred  the  sand  table  to  simulation  because  the  sand  tables  “help  lowest  level 
operators  get  a  big  picture  view”  (Nolan  &  Jones,  2005). 

Small  units  may  practice  a  tactical  scenario  by  using  a  field  or  large 
common  area  in  garrison  for  a  physical  walk-through  of  the  exercise.  The  trainer 
uses  markers  on  the  ground  to  replicate  the  terrain  from  the  scenario.  Trainees 
then  walk  along  the  replicated  routes  to  each  point  in  the  scenario,  briefing  their 
actions  as  they  would  around  the  sand  table.  Such  exercises  provide  individuals 
the  opportunity  to  visualize  what  is  happening  around  them  in  the  context  of  the 
scenario.  Such  exercises,  therefore,  provide  a  valuable  opportunity  to 
synchronize  actions  and  provide  all  participants  with  a  common  view  of  the  big 
picture. 

Returning  the  focus  to  computers,  the  gaming  landscape  changed  for  the 
individual  warfighter  in  the  past  decade,  but  the  military  did  not  provoke  it. 
Modern  personal  computers  became  powerful  enough  that  the  resulting  civilian 
gaming  market  produced  software  that  rivaled  anything  seen  in  the  defense 
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arena.  This  brings  the  discussion  back  to  1995,  to  Sergeant  Snyder  and  Marine 
Doom.  While  Sergeant  Snyder’s  project  broke  new  ground,  personal  computer- 
based  games  did  not  become  the  training  venue  of  choice  in  the  late  1990s. 
Military  culture  still  looked  at  gaming  as  a  hobby  or  recreation  rather  than 
something  to  consider  as  a  serious  training  aid.  That  culture  began  to  erode  as 
the  profile  of  computer  experience  for  the  individual  warfighter  himself  changed. 
By  2005,  a  reporter  noted  the  prevalence  of  gaming  in  Iraq.  Games  such  as  Half 
Life  2  and  Halo  competed  for  storage  space  in  the  little  deployment  libraries 
reserved  for  books  and  movies  (Associated  Press,  2005).  The  handwriting  was 
on  the  wall:  gaming  was  a  part  of  life  for  troops.  The  only  issue  was  whether  the 
military  could  glean  training  value  from  it. 

Michael  Macedonia,  a  former  Chief  Technology  Officer  for  PEO-STRI,  has 
written  about  the  change  in  military  culture  that  has  made  computer  gaming  an 
acceptable  concept.  Using  the  catch  phrase  “wired  generation,”  Macedonia 
pointed  out  that  the  pool  of  young  people  currently  filling  the  ranks  of  the  military 
has  always  known  computer  games.  He  summarized  some  U.S.  Army  studies 
by  highlighting  the  following  differences  in  the  wired  generation’s  skills  and 
attitudes; 

•  Multiprocessing,  the  ability  to  perform  several  tasks  (such  as  listen 
to  music,  talk  on  a  cell  phone,  and  use  the  computer)  concurrently 

•  Attention  span  variation  in  a  manner  similar  to  senior  executives 
exhibiting  fast  context  switching 

•  Information  navigation  changes  that  define  literacy  not  only  as  text 
but  also  as  images  and  multimedia 

•  Shift  in  focus  of  learning  from  passive  listening  to  discovery-based 
experiential  and  example-based  learning 

•  Shift  in  type  of  reasoning  from  deductive  and  abstract  to  the 
concrete 
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•  Intelligence  organized  in  easily  accessible  databases 

•  Community  of  practice  emerging  from  sharing  tasks  involving  both 
tacit  and  explicit  knowledge  over  a  substantial  period  of  time.” 
(Macedonia,  2005). 

Introduction  of  a  loose  term  is  necessary  for  the  remaining  discussion. 
Most  people  have  a  certain  mental  image  when  they  hear  the  term  “computer 
game.”  Generally,  most  people  imagine  something  played  on  a  personal 
computer  or  similar  device,  such  as  XBox.  This  concept  has  broadened  to 
include  games  that  can  be  networked  through  wire  connections  between 
computers  or  across  the  Internet.  At  the  extreme,  massively  multiplayer  games 
stretch  across  the  broad  expanses  of  the  Internet,  encompassing  thousands  of 
users.  For  the  most  part,  these  games  are  leisure  activities  both  in  design  and  in 
use.  The  military  uses  games  that  fit  into  this  general  category  for  training  or 
analysis.  To  distinguish  computer  gaming  with  an  actual  military  purpose,  Ben 
Sawyer,  a  high-tech  freelance  writer  and  technology  consultant,  coined  the  term 
“serious  games,”  and  the  term  has  caught  on  (Macedonia,  2005).  For  sake  of 
convenience,  this  thesis  incorporates  the  various  sorts  of  personal  computer- 
based  military  games  under  this  umbrella.  Serious  games  are  based  on  personal 
computers,  whether  networked  or  not.  They  are  games  in  which  participants  are 
actively  engaged  in  a  thinking  contest  against  other  participants  or  against  the 
computer  algorithms  of  the  game.  They  are  serious  in  that  they  are  intended  to 
have  military  value.  For  the  purposes  of  this  thesis,  serious  games  have  value 
for  the  direct  user;  in  other  words,  the  trainee  himself  sits  at  the  computer  and 
interacts  with  the  game. 

Serious  games  can  contribute  to  the  military.  Two  Canadian  researchers 
summarized  potential  uses  for  serious  games  in  a  paper  prepared  for  the  2008 
Interservice  /  Industry  Training,  Simulation,  and  Education  Conference: 

•  Introducing,  teaching,  and  rehearsing  new  drills; 

•  Showing  both  enemy  and  friendly  viewpoints; 
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•  Representing  the  use  and  effects  of  future  systems  not  yet 
available  for  conventional  training; 

•  Reviewing  actions  and  events  from  different  perspectives  in  post 
game  analysis,  after  action  review;  and  teambuilding  (Roman  & 
Brown,  2008). 

The  authors  further  pointed  out  that  serious  games  train  cognitive  skills 
but  not  psycho-motor  skills.  More  importantly,  the  use  of  serious  games  is  not 
just  some  theoretical  proposal  of  the  academics.  Quite  the  contrary,  serious 
games  are  already  out  there,  and  science  is  catching  up  to  see  what  operators 
are  doing  with  them. 

If  serious  games  provide  any  benefit  at  all,  their  price  tags  attract 
immediate  attention.  Serious  games  are  cheap,  particularly  if  the  civilian  market 
bears  a  parallel  research  and  development  burden.  The  past  decade  has  seen 
quite  a  bit  of  Department  of  Defense  acquisition  reform  aimed  at  minimizing  the 
massive,  stove-piped,  counterproductive  Pentagon  buying  methods.  Contractors 
with  broad  system  goals  design  and  produce,  cutting  costs  where  they  see  fit  to 
produce  the  best  product.  Serious  gaming  provides  the  perfect  arena  for  this 
type  of  acquisition.  Civilian  contractors  often  have  experience  in  designing  the 
type  of  software  the  military  needs.  The  project  simply  becomes  an  adaptation  of 
existing  ideas  to  military  requirements  (Robinson,  1998).  One  of  the  first  modern 
large-scale  efforts  toward  adapting  a  civilian  game  to  military  needs  was  Mak 
Technologies’  modification  of  Spearhead  in  the  late  1990s.  Warren  Katz,  the 
chief  executive  officer,  warned  against  oversimplifying  the  civilian  game 
adaptation  process: 

Many  people  think  you  can  take  a  video  game  out  of  a  box  and  just 
use  it  for  training  or  think  the  modifications  are  small.  The 
modifications  are  fairly  sophisticated.  Making  a  video  game  HLA- 
compliant  is  no  easy  task.  (Erwin,  2000) 

Nevertheless,  the  cost  of  the  development  of  serious  games  pales  in  comparison 
to  large-scale  simulations,  and  operational  costs  are  virtually  negligible  compared 
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to  those  associated  with  large  simulations  or  field  training.  Katz  compared 
“expensive  dome-based  simulators  with  a  motion  base  and  full  wrap-around 
imagery,”  that  cost  $5,000  to  $10,000  per  hour  to  operate,  to  a  personal 
computer-based  game  that  costs  25  cents  per  hour  (Erwin,  2000).  Clearly, 
money  talks  when  it  comes  to  serious  games. 

In  2004,  with  Operation  Iraqi  Freedom  stealing  the  spotlight  of  military 
money  and  thought,  the  Defense  Advanced  Research  Projects  Agency  (DARPA) 
fielded  DARWARS  Ambush!  to  operational  units  in  Iraq  within  six  months  of 
learning  of  a  motorized  ambush  training  shortfall.  In  the  attempt  to  use  gaming 
technology  to  mitigate  the  damage  to  operational  forces  from  ambush,  DARPA 
built  a  tactical  application  on  a  civilian  computer  game  called  Operation 
Flashpoint  (Peck,  2004).  The  game  allowed  soldiers  to  author  their  own 
scenarios  on  the  fly.  Soon,  the  Army  used  the  simulation  both  in  theater  and 
stateside  to  enhance  training.  Additionally,  personnel  began  to  experiment  with 
using  the  serious  game  for  rehearsal  with  soldiers  at  Fort  Polk,  providing  virtual 
tours  of  the  places  where  troops  would  fight  weeks  later  (Laurent,  2007).  The 
project  illustrated  that  not  only  could  serious  games  provide  training  tools 
cheaply,  but  they  could  do  it  quickly.  Just  as  important,  soldiers  liked  the  training 
and  adapted  to  it  readily.  Roger  Smith,  the  Chief  Technology  Officer  for  PEO- 
STRI,  described  soldiers  using  DARWARS  Ambush!-. 

The  soldiers  just  dive  in  and  start  ‘playing’  the  scenarios.  Then 
they  start  adapting  those  scenarios  to  make  them  more  realistic. 

They  are  not  only  learning  the  given  scenarios,  but  teaching 
themselves  to  replicate  real-life  experiences  to  re-live  and  recreate 
what  they’ve  seen  on  their  own  missions.  (McLeroy,  2008) 

A  1996  conference,  that  brought  the  military,  research,  entertainment,  and 
gaming  communities  together,  spawned  the  creation  of  a  modern-day  RAND 
Corporation  of  sorts.  The  Institute  for  Creative  Technologies  (ICT)  was  initiated 
at  the  University  of  Southern  California,  in  1999,  to  pool  the  best  ideas  of  all  the 
communities  into  collaborative  projects  for  the  benefit  of  all.  That  same  year, 

Michael  Macedonia  of  PEO  STRI  proposed  the  development  of  a  console-based 
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training  system  for  the  soldier  on  the  ground.  Five  years  later,  ICT  produced  Full 
Spectrum  Warrior.  The  game’s  title  references  the  Joint  Vision  2020  concept  of 
full-spectrum  dominance,  whereby  military  forces  control  situations  ranging  from 
relatively  peaceful  security  operations  to  the  worst  of  the  nuclear  threats.  The 
game  allows  players  to  manipulate  a  squad  through  urban  settings  to  destroy  a 
variety  of  enemy.  Full  Spectrum  Warrior  demonstrates  the  type  of  partnership 
that  serious  games  entail.  The  Army  got  a  simulation,  but  the  entertainment 
industry  got  a  top-selling  blockbuster  when  the  civilian  version  hit  the  shelves  of 
local  stores.  Full  Spectrum  Warrior  has  become  a  hit  in  the  military,  where  it  is 
even  being  considered  for  such  nontraditional  uses  as  re-creating  traumatic 
contextual  stimuli  for  treatment  of  PTSD  patients  (Halter,  2006). 

The  final  synthesis  of  leisure  and  war  training  occurred  with  America’s 
Army,  a  game  to  get  civilians  interested  in  joining  the  Army.  The  Army  sought  to 
capitalize  on  the  relationship  between  the  armchair  teenage  general  and  the 
individual  soldier  through  the  development  of  America’s  Army.  Touted  as  a 
recruiting  tool,  the  concept  of  America’s  Army  extended  well  beyond  luring  youth 
to  the  recruiting  office.  The  Army  envisioned  a  tool  that  would  attract  the  wired 
generation  to  the  military  through  leisure  activity.  Then,  the  Army  would  use  that 
same  leisure  activity  to  train  the  newly  recruited  soldiers  on  the  job.  Conceivably, 
a  young  man  would  play  on  America’s  Army  the  week  before  his  prom  and  train 
on  the  same  platform  months  later  in  preparation  for  combat.  America’s  Army  is 
unique  in  that  it  was  developed  completely  inside  the  Department  of  Defense  at 
the  Naval  Postgraduate  School’s  Modeling,  Virtual  Environments,  and  Simulation 
(MOVES)  Institute.  America’s  Army  introduced  a  new  idea  to  the  military  training 
arena:  gaming  as  a  recruiting  tool.  It  also  represented  the  first  government 
production  of  a  video  game  in  the  public  domain  (Li,  2003).  Traditionally,  the  first 
training  a  soldier  or  Marine  saw  was  marching,  shooting,  running,  and  the 
discipline  of  a  platoon  sergeant.  Now,  the  first  training  a  soldier  might  see  could 
be  a  serious  game.  The  wargame  had  potentially  gone  from  the  pastime  of  a 
bored  aristocrat  to  the  initiating  element  in  a  soldier’s  warfighting  career. 
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Serious  games  offer  another  benefit  to  the  military  constantly  on  the  go: 
deployability.  The  Marine  Corps  developed  a  program  called  Deployable  Virtual 
Training  Environment  (DVTE)  that  combined  a  variety  of  personal  computer- 
based  simulations  onto  a  single  laptop.  Suites  of  thirty  of  these  laptops  are  being 
fielded  to  each  infantry  battalion  in  the  active  and  reserve  Marine  Corps.  The 
initiative  turned  simulation  into  something  that  extended  beyond  the  physical 
bounds  of  a  camp  simulation  center  and  beyond  the  reach  of  a  technical 
contractor.  DVTE  will  be  readily  available  to  all  infantry  units  in  their  own  training 
shops.  With  a  moderately  small  number  of  computer  cases,  a  deploying 
battalion  can  pack  an  entire  simulation  center  along  for  the  trip  (Figure  1).  Units 
can  transform  static  forward  operating  bases  and  ship  berthing  into  active 
training  domains  for  the  otherwise  bored  Marine.  DVTE  makes  training  available 
anywhere  at  any  time. 


Figure  1 .  DVTE  case  for  four  computers  and  associated  equipment 
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8. 


The  Military  Does  Not  Know  Whether  Serious  Games  Work 


This  thesis  has  presented  a  series  of  serious  games  currently  used  by  the 
military  in  the  effort  to  demonstrate  the  proliferation  of  gaming  technology  in 
today’s  forces.  Examination  of  these  examples  shows  that  the  games  have 
come  into  use  primarily  because  they  are  cheap,  readily  available,  portable,  and 
have  incredible  potential  for  training,  rehearsal,  and  analysis.  Most  serious 
games  are  used  because  commanders  instinctively  recognized  their  potential 
and  started  using  them.  This  process  contrasts  with  the  acquisition  of  most 
military  systems  in  which  the  Department  of  Defense  and  contractors  go  through 
years  of  iterative  design,  testing,  and  evaluation  before  fielding  a  product.  A 
critical  review  of  Ed  Halter’s  review  of  serious  games  for  military  applications, 
entitled  From  Sun  Tzu  to  Xbox,  summed  up  the  resulting  problem  well:  “There 
just  isn’t  any  evidence  that  any  of  it  [serious  gaming  efforts  in  general]  works,  and 
Halter  doesn’t  even  bother  trying  to  prove  otherwise”  (Klein,  2006).  This 
comment  seems  to  extend  beyond  Ed  Halter;  little  research  has  been  done  to 
determine  the  effectiveness  of  serious  games.  Consequently,  these  serious 
games  are  being  used  for  serious  purposes  without  a  scientific  background 
demonstrating  their  effectiveness. 

Consider  the  case  of  Marine  Doom.  Sergeant  Snyder  produced  the  game 
in  the  attempt  to  leverage  technology  for  the  benefit  of  the  fighting  Marine.  While 
the  game  fulfilled  stated  training  objectives,  no  one  measured  whether  those 
objectives  were  truly  met,  and  the  game’s  short-lived  tenure  casts  doubt  that  they 
were.  DARWARS  Ambush!  was  produced  and  fielded  within  six  months. 
Clearly,  that  time  was  spent  on  development  rather  than  analysis  of 
effectiveness.  Even  more  fully  supported  programs  like  Full  Spectrum  Warrior 
and  America’s  Army  have  borne  little  scientific  scrutiny  into  their  effectiveness  as 
training  platforms. 

Today,  there  are  not  any  generally  accepted  methods  to  evaluate  the 
effectiveness  of  serious  games  (Roman  &  Brown,  2008).  Moreover,  there  is  no 
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accepted  practice  for  how  to  use  the  serious  games  in  training  or  analysis.  A 
close  look  at  what  operators  are  actually  doing  indicates  that  serious  games  will 
never  replace  live  training.  Rather,  they  seem  best  suited  as  training  multipliers. 
Serious  games  do  not  reduce  live  training;  they  make  it  more  effective  (Roman  & 
Brown,  2008).  Serious  games  seem  most  effective  when  used  for  the 
development  of  cognitive  skills,  such  as  decision  making  in  a  chaotic 
environment.  Users  do  not  learn  psychomotor  skills  as  well  from  serious  games. 
A  common  problem  in  serious  games  is  situational  awareness.  Serious  gaming 
has  not  been  without  research;  indeed,  a  growing  body  of  work  has  shed  some 
light  on  the  topic.  The  second  section  of  this  chapter  highlights  research  that  is 
relevant  to  this  thesis.  However,  research  thus  far  has  only  proven  that  serious 
games  offer  potential,  but  the  benefit  must  be  weighed  against  some  important 
limitations.  To  explore  these  ideas  more  fully,  the  discussion  will  turn  to  the 
specific  platform  at  the  center  of  this  research:  VBS  2™.  Before  leaving  the  more 
general  discussion,  a  summary  is  in  order. 

9.  Our  Stroll  Through  History  Tells  Us  Quite  a  Bit  About  Serious 
Games 

At  this  point,  listing  key  points  from  the  preceding  wargaming  history  is 
helpful: 

•  Wargames  have  ties  to  war  that  are  almost  as  long  as  the  history  of 
war  itself. 

•  Wargames  have  never  replaced  a  training  area  in  its  entirety;  they 
have  supplemented  training. 

•  Serious  games  have  potential  to  train  at  the  tactical,  operational, 
and  strategic  level. 

•  Serious  games  apply  in  the  cognitive  decision-making  domain. 

•  Relative  to  live  training,  serious  games  can  be  a  cheap,  readily 
available  form  of  training  any  time  and  any  place. 
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•  Serious  games  appeal  to  the  new  warfighter  coming  from  the  wired 
generation. 

•  Serious  games  can  tie  the  military  to  the  civilian  world  and  vice 
versa. 

10.  VBS  2™  is  the  Marine  Corps’  Bid  to  Capitaiize  on  the  Potentiai 
of  Serious  Games 

The  Marine  Corps  has  moved  forward  with  an  initiative  that  illustrates  the 
potential  of  serious  games.  With  the  relative  success  of  Marine  Doom,  the 
Marine  Corps  sought  a  commercial  gaming  product  that  would  enable  tactical 
training  across  a  wide  range  of  warfighting  domains.  Bohemia  Interactive,  an 
Australian  company,  manufactured  Operation  Flashpoint,  a  commercial  game 
that  could  be  modified  to  fit  the  service’s  needs.  The  Marine  Corps  reviewed  the 
game  in  depth  and  issued  a  statement  of  work  to  modify  the  game  into  a  realistic 
Marine  environment  with  exclusive  rights  for  future  modifications  as  necessary. 
VBS  2™  resulted  from  the  request.  VBS  2™  is  a  first-person  shooter  game  that 
can  be  networked  to  involved  many  players  at  once.  Using  a  drag-and-drop 
menu,  game  administrators  can  develop  virtually  any  realistic  scenario  on  the 
various  terrain  databases  available  in  the  system.  Game  versatility  easily  allows 
players  to  drive  armed  trucks  through  the  desert,  patrol  dismounted  in  town,  work 
with  tanks,  artillery,  and  aircraft  in  the  open  field,  or  interact  on  a  limited  basis 
with  civilians  and  coalition  partners.  While  various  games  may  outperform  VBS 
2™  in  different  areas,  VBS  2™  offers  the  advantages  discussed  above  to 
individual  Marines  at  their  personal  computer  or  battalion  laptop. 

However  well  VBS  2™  might  look  on  the  specification  sheet,  little  has 
been  done  to  demonstrate  or  prove  that  it  actually  works  as  a  training  device.  To 
discuss  this  issue  intelligently,  one  first  must  establish  what  VBS  2™  is  actually 
supposed  to  do.  The  Marine  Corps  sought  a  virtual  training  platform  for 
individual  Marines  for  all  the  reasons  simulations  such  as  Full  Spectrum  Warrior 
were  developed.  Marines  were  already  into  gaming  from  their  pre-service  years. 
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serious  games  were  cheap,  serious  games  were  deployable,  serious  games 
were  easy  to  administer.  The  statement  of  work  for  VBS  2™  was  based  on  a 
cognitive  task  analysis  completed  in  2004.  Program  Manager  Training  Systems 
(PM  TRASYS)  contracted  with  CHI  Systems,  Inc.  and  Klein  Associates,  Inc.  to  do 
something  best  explained  by  the  report’s  title  “Using  cognitive  task  analysis  to 
support  cognitive  authenticity  in  training  strategies  for  anti-terrorism  force 
protection  tactical  decision  making”  (J.  Styer,  personal  communication,  October 
27,  2008,  and  P.  Nichols,  personal  communication,  November  12,  2008). 
Operation  Flashpoint  fulfilled  several  goals  quickly.  It  was  the  right  size  in  terms 
of  memory.  It  provided  the  right  amount  of  versatility  and  fidelity.  Its  flexible  and 
friendly  mission  editor  was  deemed  critically  important.  Most  importantly,  it  was 
close  to  what  the  Marine  Corps  needed,  requiring  the  least  amount  of 
modification.  As  a  result.  Operation  Flashpoint  was  the  game  of  choice,  and 
VBS  2™  was  born.  The  only  official  document  linking  the  game  to  performance 
standards  was  the  PM  TRASYS  cognitive  task  analysis  (D.  Mathes,  personal 
communication,  October  27,  2008). 

The  cognitive  task  analysis  did  specify  performance  criteria  in  measurable 
Marine  Corps  terms.  At  the  time,  the  Marine  Corps  used  a  system  of  “Individual 
Training  Standards”  (ITS)  that  specified  required  tasks  and  the  precise 
measurable  steps  to  perform  those  tasks  to  standard.  The  cognitive  task 
analysis  listed  the  ITSs  required  for  the  missions  they  analyzed.  The  cognitive 
task  analysis  focused  on  the  Fleet  Anti-terrorism  Security  Team  (FAST) 
company,  an  entity  representing  a  relatively  narrow,  unique  skill  set. 
Generalizing  this  task  list  to  the  entire  Marine  Corps  certainly  shortchanges  the 
capabilities  of  VBS  2™.  The  game  can  do  so  much  more.  Thus,  identifying  what 
VBS  2™  is  actually  supposed  to  do  is  muddy  water,  and  the  available 
documentation  simply  leaves  one  making  things  up  based  on  intuition. 

VBS  2™  has  experienced  some  scientific  scrutiny,  although  it  has  come 
from  outside  the  United  States.  In  2002,  some  Australian  computer  science 
researchers  at  the  University  of  New  South  Wales  began  to  explore  Operation 
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Flashpoint  as  a  tactical  training  tool.  They  evaluated  users  in  a  series  of  three 
exercises  in  the  game.  From  survey  results,  they  noted  such  problems  as 
disorientation  from  the  small  screen  size,  lack  of  simulated  fatigue,  and 
unrealistic  tactics  among  computer-generated  forces.  Players  could  not  practice 
basic  motor  skills  such  as  weapons  drills.  Nevertheless,  they  showed  that  the 
game  could  be  used  for  tactical  training  with  limitations  (Barlow,  Morrison,  & 
Easton,  2002).  Once  the  VBS  project  had  matured  somewhat,  in  2004,  the  same 
Australian  research  team  reviewed  the  game  system  again.  They  continued  to 
rate  the  game  as  a  good  tool  for  section  level  training,  although  situation 
awareness  issues  still  topped  the  list  of  limitations.  The  AAR  capability 
requested  by  the  Marine  Corps  added  significant  capability  to  the  game 
(Morrison  &  Barlow,  2004).  In  the  same  year,  one  of  the  researchers  worked 
with  a  group  at  the  Virtual  Environments  &  Simulation  Laboratory  (VESL)  to  study 
how  squad  leaders  make  decisions  under  the  stress  of  battle.  The  team 
successfully  demonstrated  that  serious  games  can  be  used  for  analysis  as  well 
as  training  (Barlow,  Luck,  Lewis,  Ford,  &  Cox,  2004).  The  next  year,  the  team 
highlighted  a  striking  limitation  of  VBS  2™  and  serious  games  in  general:  that  is, 
players  must  practice  with  the  game  for  periods  ranging  from  a  few  hours,  for 
basic  skills,  to  a  week  or  more  for  more  complex  team  leading  tasks  in  order  to 
use  the  game  seamlessly  as  an  extension  of  their  warfighting  skills  (Morrison, 
Barlow,  Bethel,  &  Clothier,  2005).  All  of  this  research  supported  the  murkiness  of 
the  situation:  serious  games  have  potential  as  a  training  tool,  analytical  asset, 
and  rehearsal  mechanism,  but  these  benefits  come  with  notable  limitations. 
Clearly,  the  research  suggested  the  need  for  a  closer  look  at  tactical 
performance  on  the  ground  after  game  training. 

More  recently,  the  Canadian  Combat  Training  Centre  demonstrated  some 
degree  of  effectiveness  for  VBS  2™.  They  incorporated  the  serious  game  into 
their  Troop  Warrant  Officer’s  course,  reducing  live  field  training  while  increasing 
performance  results.  Their  experimentation  resulted  in  very  specific  data 
showing  that  use  of  VBS  2™  resulted  in  a  more  effective  blended  training 
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program,  but  the  effort  was  not  structured  in  a  way  that  would  show  specifically 
which  aspects  of  the  serious  game  worked  and  which  did  not.  For  one  serial  of 
the  course,  the  school  used  1  day  of  VBS  2™  training  and  5.5  weeks  of  live 
training  in  the  field.  Noting  success  from  use  of  the  simulation,  the  school  turned 
2.5  weeks  of  the  live  training  into  VBS  2™  training  and  noticed  a  significant 
improvement  in  success  rate  in  field  exercises.  However,  because  the  school 
was  merely  getting  training  done  and  not  running  a  formal  experiment,  these 
results  were  tainted  with  confounds,  such  as  changes  in  instructor  cadre  (Roman 
&  Brown,  2008). 

The  background  study  thus  far  has  demonstrated  the  potential  of  serious 
games,  and  that  VBS  2™  is  well  designed  to  realize  that  potential.  While  VBS 
2™  has  been  examined  in  the  scientific  and  academic  arenas,  however,  nothing 
but  anecdotal  evidence  exists  to  demonstrate  its  effectiveness.  Intuition  and 
research  thus  far  indicate  that  VBS  2™  has  much  to  offer  the  individual  Marine. 
Clearly,  a  glimpse  into  this  effectiveness,  or  lack  thereof,  is  warranted.  The  next 
section  examines  how  this  might  be  accomplished. 

C.  TRAINING  TRANSFER 

Training  transfer  is  the  idea  that  training  produces  results  in  the  form  of 
trainee  performance  improvement  as  intended.  It  seems  simple  enough,  on  the 
surface,  to  test:  just  get  two  groups  of  people,  train  one  group  the  old-fashioned 
way  and  the  other  group  using  the  method  under  investigation,  and  compare  the 
results.  Intuition  dictates  a  few  complexities,  such  as  the  need  for  measurable 
results  and  the  standardization  of  peripheral  factors  so  that  the  effects  of  the  new 
training  method  can  be  isolated.  However,  training  transfer  studies  are  not 
necessarily  as  straightforward  as  they  may  seem,  and  an  examination  of  past 
studies  and  related  research  sheds  some  light  on  the  difficulties. 

Several  combinations  of  words  exist  in  conjunction  with  the  word  “transfer” 
that  revolve  around  a  similar  group  of  concepts.  For  the  purposes  of  this  thesis, 
it  is  worth  taking  a  closer  look  at  two  of  these  word  groupings.  Transfer  of 
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training,  which  is  assumed  to  be  the  same  as  training  transfer,  is  defined  as  “the 
extent  of  retention  and  application  of  knowledge,  skills,  and  attitudes  from  the 
training  environment  to  the  workplace  environment”  (Bossard,  Kermarrec,  Buche, 
&  Tisseau,  2008).  Transfer  of  learning  is  a  slightly  different  concept  that  applies 
to  the  educational  arena.  If  transfer  of  learning  is  accomplished,  a  student  can 
generalize  something  learned  in  the  classroom  to  broader  application  areas,  so 
that  the  individual  uses  knowledge  gained  from  the  learning  process  to  solve 
problems  that  have  not  been  presented  before.  As  a  rule  of  thumb,  transfer  of 
training  generally  applies  to  work  and  transfer  of  learning  applies  to  education 
(Bossard  et  al.,  2008).  Both  concepts  share  the  idea  that  knowledge  and  skills 
can  be  applied  in  a  practical,  measurable  environment. 

In  virtual  environments,  a  concept  appears  throughout  relevant  literature 
that  differentiates  between  transfer  related  directly  to  the  training  context  and 
transfer  generalized  beyond  the  context.  Vertical  transfer  refers  to  the  ability  of 
the  learner  to  recognize  elements  of  the  training  context  and  apply  what  he  or 
she  has  learned  to  problems  of  increasing  complexity.  Horizontal  transfer,  on  the 
other  hand,  describes  the  learner’s  application  of  knowledge  and  skills  gained 
through  the  training  to  general  problems  that  extend  beyond  the  immediate 
context  of  what  was  taught.  Other  terms,  such  as  “near”  and  “far”  or  “general” 
and  “specific”  have  slightly  different  meanings  but  hinge  on  the  same  idea. 
Whatever  the  terminology,  it  is  important  to  keep  in  mind  that  transfer  of  largely 
procedural  knowledge  in  a  set  list  of  clearly  defined  steps  is  much  different  than 
transfer  of  generalized  situational  knowledge  applied  in  varying  circumstances. 
Moreover,  context  is  considered  critical;  some  argue  that  context  is  so  important 
that  straying  from  the  original  context  eliminates  the  possibility  of  transfer 
(Bossard  et  al.,  2008). 

1.  The  Trainee  Matters 

People  learn  in  different  ways,  and  these  differences  affect  the  level  of 
transfer.  Cognitive  ability  affects  the  amount  of  transfer,  with  one  study 
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concluding  that  this  factor  accounts  for  as  much  as  16%  of  the  variance  in 
training  effectiveness.  Self-efficacy  has  its  own  effect;  if  an  individual  does  not 
feel  competent  to  perform  a  task,  the  training  transfer  will  quite  likely  suffer. 
Motivation  can  affect  transfer  in  a  couple  of  ways.  An  individual  may  or  may  not 
be  motivated  to  train,  influencing  the  effectiveness  of  the  effort.  An  individual 
may  not  be  motivated  to  use  the  skills  in  the  real  world,  an  issue  called 
motivation  to  transfer.  Personality  traits  can  affect  transfer;  among  those  studied 
with  transfer  impact  are  anxiety,  openness  to  experience,  extroversion,  and 
conscientiousness.  Training  transfer  tends  to  be  maximized  when  the  individual 
perceives  that  the  knowledge  and  skills  being  trained  will  improve  a  relevant 
aspect  of  his  or  her  work  performance.  Training  transfers  better  with  increases  in 
the  trainee’s  job  involvement.  Training  transfer  also  depends  on  the  degree  to 
which  the  individual  identifies  with  his  job  and  considers  improvements  in 
performance  important  to  his  self  worth.  Science  has  not  yet  shown  the  perfect 
profile  of  the  individual  who  will  easily  transfer  training  or  the  individual  who  will 
struggle  with  it,  but  the  science  is  clear  that  transfer  will  differ  from  person  to 
person.  Some  key  characteristics,  such  as  cognitive  ability,  pretraining 
motivation,  negative  affectivity,  and  perceived  utility,  may  help  identify  those 
individuals  who  will  struggle  in  a  training  transfer  endeavor.  More  importantly, 
one  may  be  able  to  take  actions  in  the  training  phase  to  minimize  the  effects  of 
some  of  these  characteristics,  such  as  negative  affectivity  or  perceived  utility 
(Burke  &  Hutchins,  2007). 

This  thesis  does  not  seek  to  expound  in  depth  on  the  various  schools  of 
thought  in  learning  psychology,  but  it  is  helpful  to  quickly  highlight  the  three  most 
common  theories:  behavioral  learning,  cognitive  learning,  and  constructive 
learning.  Psychologists  adhering  to  the  behavioral  approach  contend  that  the 
human  mind  is  a  sort  of  black  box,  where  inputs  can  be  studied  and  outputs  can 
be  studied,  but  the  mental  processes  in  the  middle  are  ignored.  Key  side  notes 
from  this  school  of  thought  are  the  idea  that  the  learner  adapts  to  the 
environment  and  the  concept  of  learning  as  a  largely  passive  process. 
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Proponents  of  cognitive  learning,  on  the  other  hand,  focus  on  the  black  box  itself, 
attempting  to  model  what  goes  on  inside  the  learner’s  brain.  In  cognitive 
learning,  the  individual  conceptually  has  a  knowledge  base  and  uses  processes 
and  symbols  to  map  what  is  taught  into  the  base.  Thus,  study  of  this  approach 
seeks  to  identify  and  describe  those  processes  and  symbols.  Both  behavioral 
and  cognitive  learning  approaches  view  the  learning  process  as  one  in  which  the 
knowledge  presented  is  set  and  absolute,  like  a  traditional  classroom  lecture 
environment.  Psychologists  adhering  to  the  constructivist  school  of  thought 
challenge  this  idea,  contending  that  each  learner  reconstructs  the  facts 
presented  in  his  or  her  own  way  to  build  the  internal  knowledge  base.  Like  the 
cognitive  approach,  knowledge  is  viewed  as  a  combination  of  symbols  and 
processes  to  access  a  knowledge  base  in  the  head,  but  the  constructivist 
approach  holds  that  each  individual  has  his  or  her  own  world  perspective  based 
on  individual  experience  (Bossard  et  al.,  2008). 

Military  training  often  reflects  a  behavioral  approach  out  of  necessity. 
Training  typically  occurs  en  masse,  and  proficiency  is  achieved  through 
repetition,  discipline,  and  remediation.  The  situation  changes  when  training 
orients  on  decision  making.  In  a  platoon  training  event,  a  junior  Marine  may 
experience  a  primarily  behavioral  training  regimen  while  the  platoon  commander 
experiences  a  more  cognitive  or  constructivist  approach.  As  pointed  out  in  the 
training  transfer  introductory  section,  transfer  itself  may  be  contextually  specific 
or  more  generalized.  These  taxonomies  become  important  to  the  discussion  of 
unit  level  training.  In  the  team  environment,  some  individuals  may  be  learning 
very  lock-step,  procedural  tasks  while  others  may  be  learning  very  abstract, 
generalized  decision-making  schemes  that  will  most  likely  be  applied  in  new  and 
unique  situations.  When  the  overall  transfer  of  group  training  is  under  the 
microscope,  one  must  bear  in  mind  the  different  training  models  in  place  at  the 
individual  level. 

When  technology  is  used  for  training,  the  trainee’s  attitude  toward  that 
technology  can  influence  the  training  experience.  A  U.S.  Army  project  explored 
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this  idea.  A  first-year  class  of  cadets  at  the  U.S.  Military  Academy  trained  with 
America’s  Army,  a  game  discussed  in  the  first  section  of  this  chapter.  The 
research  team  sought  to  determine  whether  gaming  experience  and  computer 
self-efficacy  affected  four  learner  outcomes  of  interest:  training  motivation, 
training  satisfaction,  ease  in  using  the  game  interface,  and  perceived  team 
cohesion.  Not  only  did  the  research  team  find  that  these  two  factors  affected 
learner  outcomes,  but  they  found  that  the  type  of  experience  mattered  as  well. 
For  example,  gamers  who  used  products  similar  to  America’s  Army  had  higher 
training  motivation,  training  satisfaction,  ease  in  using  the  interface,  and 
perceived  team  cohesion  than  gamers  who  played  much  different  games  (Orvis, 
Orvis,  Belanich,  &  Mullin,  2005).  As  pointed  out  above,  a  trainee’s  learning  style 
matters,  but  a  trainee’s  experience  with  computer  technology  in  general,  and 
experience  with  technology  similar  to  that  used  for  training,  makes  a  difference 
as  well. 

2.  The  Way  Training  Is  Done  Matters 

The  design  and  delivery  of  the  training  intervention  can  affect  transfer. 
Before  the  training  is  planned,  a  needs  analysis  can  determine  whether  it  is 
needed.  If  the  training  is  not  necessary,  training  transfer  results  will  obviously 
suffer  accordingly.  During  the  training  intervention,  establishment  of  learning 
goals  and  objectives  positively  impacts  training  transfer;  that  is,  people  learn 
better  if  they  know  what  is  expected  of  them  in  the  course  of  the  training. 
Content  relevance  is  also  important;  trainees  must  see  a  relationship  between 
training  content  and  work  tasks. 

The  way  the  material  is  taught  influences  training  transfer.  Practice  and 
feedback  provide  the  opportunity  for  trainees  to  reinforce  what  has  been 
presented,  enhancing  transfer  in  the  long  run.  Overlearning,  the  practice  of 
having  individuals  continue  practicing  even  after  they  have  correctly 
demonstrated  the  skill,  can  improve  transfer  of  training  by  making  responses 
automatic.  Trainees  can  experience  cognitive  overload,  in  which  they  are 
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presented  more  than  they  can  learn  at  one  time.  Strategies  that  minimize 
information  not  necessary  for  learning,  while  maximizing  the  information  that 
directly  contributes  to  learning,  improve  training  transfer.  Active  learning  quite 
likely  affects  transfer  more  positively  than  lecture,  although  the  literature  is 
inconclusive  on  this  point.  Error-based  examples  help  trainees  learn  a  skill  by 
showing  aspects  of  the  task  that  can  go  wrong,  and  such  strategies  have  been 
found  to  increase  training  transfer  (Bossard  et  al.,  2008). 

A  common  workplace  phenomenon  is  the  individual  returning  from  a 
training  workshop  with  the  newest  way  of  doing  something.  Quite  likely,  that 
individual  returns  to  a  skeptical  crowd,  unwilling  to  bother  with  changes  to  the 
status  quo.  In  the  end,  the  individual’s  training  might  end  up  useless.  The 
individual  may  have  learned  the  knowledge  and  skills  perfectly.  He  or  she  may 
have  been  able  to  apply  them  any  time  and  any  place.  The  individual  may  have 
returned  with  a  burning  enthusiasm  to  apply  the  knowledge  and  skills 
immediately.  Nevertheless,  all  of  this  failed  in  the  face  of  resistance  back  in  the 
office. 

The  organization  itself  is  critical  to  training  transfer.  For  example,  training 
transfer  improves  if  the  trainee  perceives  a  direct  link  between  the  training  and 
the  supervisor’s  strategic  goals.  The  propensity  for  the  organizational 
atmosphere  to  contribute  to  training  transfer  is  described  by  the  term  “transfer 
climate.”  A  positive  transfer  climate  encourages  use  of  the  newly  trained  skills 
and  incentivizes  their  correct  application.  Supervisory  support  impacts  training 
transfer.  The  support  of  peers  is  even  more  important  than  the  support  of 
supervisors.  If  a  skill  and  applied  training  technique  gain  acceptance  at  the  water 
cooler,  transfer  improves.  The  organization  also  contributes  to  positive  training 
transfer  by  affording  the  opportunity  to  perform  the  newly  acquired  skills.  The 
organization’s  methods  and  pressure  to  hold  individuals  accountable  to  what  they 
have  learned  seems  to  affect  training  transfer  as  well,  although  the  available 
research  fails  to  describe  this  link  in  detail  (Bossard  et  al.,  2008). 
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3. 


Numbers  Matter 


The  practical  matter  of  assessing  training  transfer  is  based  on  quantifying 
results  of  some  job  or  skill  performance.  Ultimately,  performance  must  be 
translated  into  a  number  or  combination  of  numbers  that  can  be  statistically 
compared  and  analyzed.  Performance  measurement  is  a  science  all  of  its  own, 
and  the  problems  are  exacerbated  when  reviewing  performance  of  teams  instead 
of  performance  of  individuals.  Jack  Zigon  presides  over  a  consultant  firm 
specializing  in  team  performance  measurement  systems.  He  highlighted  three 
reasons  team  performance  measurement  is  difficult: 

•  It  is  not  always  obvious  what  results  should  be  measured. 

•  Even  if  you  know  what  to  measure,  it  is  often  not  clear  how  the 
measurement  should  be  done. 

•  Teams  are  made  up  of  individuals,  thus  measurement  must  be 
done  at  both  the  team  and  individual  levels. 

Zigon’s  work  oriented  on  the  corporate  world,  but  he  offered  some  advice 
that  is  useful  for  any  performance  measurement.  First,  he  recommended 
focusing  on  results,  because  data  related  to  the  activity  that  produced  the  results 
can  be  misleading  or  uninformative.  Second,  he  discussed  the  creation  of 
measures  for  each  accomplishment,  noting  that  measures  cannot  always  be 
numerical.  He  categorized  measures  into  numeric  measures  that  use 
quantification  of  some  concrete  aspect  of  the  accomplishment  and  descriptive 
measures  that  use  words  for  evaluation.  Third,  he  recommended  developing  a 
system  of  performance  standards.  These  standards  use  the  measures  from  the 
second  step  to  determine  how  a  team  performs  against  a  set  goal.  Finally,  Zigon 
suggested  the  implementation  of  a  feedback  system.  The  feedback  system 
provides  the  individual,  the  team,  and  the  organization  an  assessment  of  the 
performance  (Zigon,  1998). 

The  project  that  is  the  subject  of  this  thesis  seeks  to  capitalize  on  years  of 
Marine  Corps  experience  in  team  performance  evaluation  to  prevent  a 
reinvention  of  the  wheel.  The  Marine  Corps  Mission  Performance  Standards 
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(MPSs)  list  all  of  the  skills  Marines  in  a  particular  job  specialty  should  perform  to 
be  proficient  within  that  specialty  at  each  rank  of  service.  Marine  Corps  Order 
(MCO)  P3500.72A,  dated  18  April  2005,  outlines  the  training  and  readiness 
program  that  incorporates  these  MPSs  into  a  training  regimen  for  ground  forces. 
In  the  publication’s  introduction  to  Marine  Corps  training  philosophy,  MCO 
P3500.72A  explains  in  very  direct  terms  how  the  service  views  training  and 
training  evaluation:  “Training  Marines  to  perform  as  a  team  in  combat  lies  at  the 
heart  of  the  Training  and  Readiness  (T&R)  Program.  Unit  readiness  and 
individual  readiness  are  directly  related.”  The  publication  then  specifies  the 
tenets  on  which  the  T&R  concept  is  built: 

•  Focus  on  expected  combat  missions 

•  Building  block  approach  to  training 

•  Focus  on  Individual  Core  Skills  and  Unit  Core  Capabilities 

•  Organization  of  tasks  into  executable  events 

•  Sustainment  of  training 

MCO  P3500.72A  addresses  core  skills,  which  apply  to  the  individual,  and 
core  capabilities,  which  apply  to  the  section  or  unit.  An  individual  must  be  able  to 
perform  core  skills  to  be  qualified  for  his  or  her  job  specialty.  A  section  or  unit 
must  be  able  to  perform  core  capabilities  in  order  to  meet  performance 
expectations  in  contingency  operations  or  combat.  Closely  related  to  these 
concepts  are  Mission  Essential  Tasks  (METs)  which  are  those  tasks  that  form 
“the  very  essence  of  the  community’s  existence.”  Skills  that  are  “environment, 
mission,  rank,  or  billet  specific”  are  core  plus  skills,  and  advanced  functions  that 
are  “environment,  mission,  or  theater  specific”  are  core  plus  capabilities.  In  order 
to  understand  the  organization  of  the  MPSs,  one  must  keep  in  mind  the  building 
block  approach  used  by  the  training  and  readiness  program.  The  program 
organizes  the  standards  as  executable  events  that  units  can  arrange  into 
appropriate  field  exercises  and  training  programs.  Some  standards  serve  as 
prerequisites  for  others,  and  units  can  use  chaining  to  give  sustainment  credit  for 
simpler  tasks  that  support  more  advanced  events.  In  this  way,  the  training  and 
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readiness  program  serves  as  the  basis  for  school  syllabuses  as  well  as 
peacetime  training  in  operational  units.  The  same  program  that  defines 
requirements  for  entry-level  job  specialty  training  also  specifies  how  individuals  in 
that  job  specialty  will  train  throughout  their  careers  and  how  their  units  will  train 
as  well. 

The  training  and  readiness  program  evaluates  training  in  two  ways. 
Proficiency  measures  performance  of  a  certain  skill  or  set  of  skills  against  a  set 
standard.  Currency  evaluates  the  standard  against  a  sustainment  interval  for  the 
particular  event.  Thus,  training  may  be  necessary  because  an  individual  or  unit 
cannot  perform  a  skill  well  enough,  or  because  too  much  time  has  elapsed  since 
the  skill  was  last  performed. 

The  administration  of  the  training  and  readiness  program  supports 
organizing  and  analyzing  training  events.  Each  MPS  is  coded  with  a  series  of 
three  four-character  codes.  The  first  four-character  grouping  indicates  the  job 
specialty,  the  second  four-character  grouping  indicates  the  functional  area  or 
duty  area,  and  the  third  four-character  grouping  indicates  the  level  and 
sequence.  The  program  ultimately  seeks  to  measure  a  unit’s  preparation  for 
each  of  its  METs.  The  Combat  Readiness  Percentage  (CRP)  quantifies  this 
preparation.  Specific  events  that  support  a  MET  for  CRP  calculation  are  called 
evaluation  coded  events  or  E-coded  events.  Thus,  E-coded  events  are  those 
that  contribute  to  an  overall  grade  or  numerical  evaluation. 

Each  MPS  has  a  title  and  a  description  that  explains  the  purpose, 

objectives,  goals,  and  requirements  of  the  event.  The  event’s  condition  lists  the 

items,  such  as  equipment,  manuals,  tools,  and  aids,  that  must  be  provided  as 

well  as  any  specific  conditions  under  which  the  event  must  be  performed.  The 

event’s  standard  explains  the  minimum  acceptable  level  of  performance  of  the 

event  and  how  the  level  of  performance  will  be  judged.  Each  event  has  a  list  of 

performance  steps  that  guide  an  evaluator  to  ensure  all  components  of  the  MPS 

have  been  satisfied.  An  event  may  have  a  list  of  prerequisites  or  chained  events. 

Finally,  each  event  has  a  list  of  references  and  support  requirements.  For  each 
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task  supporting  a  unit’s  mission,  the  training  and  readiness  system  specifies  what 
the  task  is,  how  to  do  it,  how  to  measure  it,  and  how  to  support  it. 

Navy  and  Marine  Corps  Publication  (NAVMC)  3500.87  is  the  Marine 
Corps’  infantry  training  and  readiness  manual.  With  more  than  600  pages  of 
documentation,  the  publication  explicitly  states  how  the  Marine  Corps  infantry 
should  train,  from  the  individual  riflemen  through  the  large  battalion  and 
regimental  staffs.  NAVMC  3500.87  meticulously  follows  the  guidelines  of  the 
training  and  readiness  program,  providing  a  highly  detailed  and  organized 
method  of  delineating  what  tasks  must  be  accomplished  for  any  given  mission, 
how  they  will  be  accomplished,  and  how  they  can  be  graded. 

4.  Determining  What  Matters 

Lisa  Burke  and  Holly  Hutchins  (2007)  collaborated  on  an  integrative 
literature  review  of  training  transfer.  In  summarizing  available  research,  they 
recommended  that  training  transfer  be  assessed  as  “a  multidimensional 
phenomenon  with  multilevel  influences.”  This  concept  is  probably  the  most 
critical  idea  related  to  training  transfer.  Transfer  is  not  a  black  and  white  switch 
that  either  happens  or  fails  to  occur.  Rather,  the  individual  trainee,  the  training 
plan,  and  the  training  environment  all  have  many  variables  working  together  to 
influence  the  degree  of  transfer.  Understanding  these  potential  variables  allows 
one  to  try  to  minimize  their  impact  but,  in  the  end,  many  factors  can  affect  the 
final  transfer  of  training. 

Applicable  literature  recognizes  this  multi-factored  aspect  of  training 
transfer.  One  model  proposed  to  deal  with  the  issue  is  the  generalized  Learning 
Transfer  System  Inventory.  LTSI  attempted  to  design  a  scaling  system  for 
measuring  transfer  based  on  various  factors.  The  generalized  study  included 
sixteen  factors  analyzed  via  questionnaire.  While  LTSI  took  individual  level 
training  transfer  a  step  too  far  for  this  thesis,  an  overview  of  the  sixteen  factors  is 
useful.  Some  of  the  factors  focused  on  the  individual,  such  as  learner  readiness, 
motivation  to  transfer,  positive  and  negative  personal  outcomes,  and  personal 
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capacity  for  transfer.  Other  factors  focused  on  the  environment  or  climate  such 
as  peer  support,  supervisor  support,  supervisor  sanctions,  performance 
coaching,  and  opportunity  to  use.  Several  factors  related  to  the  way  the 
individual  perceives  the  training  affecting  him  or  her,  such  as  perceived  content 
validity,  performance  expectations,  performance  outcomes  expectations,  and 
performance  self-efficacy.  Finally,  the  transfer  design  itself  affects  transfer. 
When  researchers  study  training  transfer,  reporting  information  about  the 
circumstances  of  the  training  experiment  using  these  factors  can  help 
standardize  studies  in  the  aggregate.  More  practically,  though,  the  LTSI  model 
itself  can  help  the  training  organization  as  an  indicator  of  transfer  inhibitors  so 
that  they  can  be  minimized.  For  the  purposes  of  a  training  transfer  study  like  the 
one  proposed  in  this  thesis,  LTSI  provided  a  sort  of  guideline  for  factors  that 
need  to  be  controlled  throughout  the  study  in  order  to  isolate  the  effects  of  the 
training  mechanism  itself  (Holton,  Bates,  &  Ruona,  2000). 

In  summary,  training  transfer  depends  on  many  factors,  and  any  attempt 
to  measure  transfer  must  take  this  into  account.  A  list  of  pertinent  notes  from  this 
study  follows: 

•  Trainees  learn  differently. 

•  Transfer  improves  if  the  trainees  know  what  they  are  doing  and 
why. 

•  The  training  climate  can  positively  or  negatively  impact  training. 

While  the  Marine  Corps  has  a  ready-made  system  of  evaluating  training,  these 
factors  proved  important  throughout  the  course  of  the  project. 

5.  A  Tribute  to  Those  Who  Have  Gone  Before 

Simulator  use  without  robust  training  transfer  research  is  prevalent  across 
a  wide  variety  of  training  domains,  including  medical  procedures  training,  nuclear 
power  plant  operation,  commercial  aviation,  and  the  NASA  space  program.  The 
bottom  line  is  that  training  transfer  research  is  hard.  Human  performance  is 
difficult  to  measure,  and  the  difficulties  expand  when  the  exercise  involves 
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teams.  Additionally,  training  is  quite  likely  not  the  sole  contributor  to  subsequent 
proficiency.  A  proper  evaluation  must  isolate  training  from  other  factors  such  as 
environment,  leadership,  and  equipment  (Thurman  &  Dunlap,  1999). 
Nevertheless,  some  research  projects  have  evaluated  the  training  transfer  of 
personal  computer-based  training.  While  the  following  list  does  not  strive  to  be 
comprehensive,  it  demonstrates  some  of  the  efforts  that  have  been  made  and 
the  successes  and  challenges  encountered  along  the  way. 

A  research  project  at  the  Israeli  Air  Force  flight  school  used  a  personal 
computer-based  game  called  Space  Fortress  II  to  help  train  cadet  pilots.  Two 
groups  of  cadets  received  ten  hours  of  simulation-based  training  while  a  control 
group  received  traditional  training.  The  effects  of  the  training  were  evaluated  in  a 
series  of  eight  flights  that  were  part  of  the  training  program.  Flight  instructors 
graded  the  cadets’  performance  on  several  criteria  with  numerical  markings  and 
then  gave  the  students  an  overall  score.  Comparison  of  the  results  indicated  that 
both  simulation  groups  performed  better  in  the  live  flight  exercises  than  the 
control  group.  One  of  the  simulation  groups  trained  using  single  task  games. 
These  part-task  trainees  trained  with  one  part-task  game  at  a  time  with  feedback 
on  each  game.  Afterward,  the  trainees  played  the  whole  game.  The  other  group 
trained  with  the  whole  game  only,  receiving  feedback  at  the  end.  The 
researchers  found  that  the  trainees  who  had  used  the  part-task  games 
outperformed  the  cadets  who  trained  with  the  full  game  (Gopher,  Weil,  & 
Bareket,  1994).  This  study  shows  how  a  personal  computer-based  game  can 
result  in  enhanced  performance  in  the  real  world. 

In  1998,  a  British  study  explored  the  use  of  Microsoft  Flight  Simulator  to 
train  pilots.  This  study  featured  two  simulation  groups:  one  group  completed  the 
simulation  using  controls  similar  to  those  in  an  airplane,  while  the  other  group 
used  keyboard  and  mouse  controls.  A  control  group  trained  using  traditional 
means.  Students  then  flew  real-world  flights  after  the  training,  and  flight 
instructors  rated  designated  skills  on  five-point  scales.  The  study  determined 
that  both  simulation-trained  groups  performed  better  than  the  control  group. 
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Also,  the  group  trained  in  the  simulation  with  controls  that  were  similar  to  those  in 
an  airplane  performed  better  than  those  who  used  the  keyboard  and  mouse 
interface.  The  researchers  concluded  that  a  low-resolution  trainer  such  as  a 
personal  computer-based  game  could  effectively  contribute  to  training  in  the 
early  stages  of  a  pilot’s  career.  The  authors  were  careful  to  point  out  that  their 
results  did  not  suggest  that  personal  computer-based  simulations  aided  in 
psychomotor  skill  acquisition.  Also,  the  study  was  limited  to  a  relatively  simple 
flight  task  of  flying  a  square  where  subtasks  included  flying  straight  and  level  and 
entering  and  exiting  turns.  The  study  could  not  generalize  its  findings  to  more 
complex  tasks  like  taking  off  and  landing  (Dennis  &  Harris,  1998).  However,  this 
study  shows  another  definitive  instance  of  personal  computer-based  training 
enhancing  real-world  performance. 

A  2007  study  at  Bristow  Academy,  with  a  commercial  personal  computer- 
based  game  called  X-Plane,  sought  to  determine  whether  personal  computer- 
based  gaming  technology  could  be  used  for  helicopter  training.  The  experiment 
involved  three  groups,  differing  by  interface:  those  in  a  mock  cabin  with  motion 
feedback,  those  in  a  mock  cabin  without  motion  feedback,  and  those  using  a 
desktop  interface.  The  researchers  measured  training  transfer  by  determining 
whether  each  group  showed  improvement  in  the  simulator.  Analysis  of 
experiment  results  yielded  conclusions  that  both  cabin  arrangements  provided 
training  transfer  while  desktop  configurations  resulted  in  no  significant  transfer. 
Also,  experiment  results  indicated  that  motion  did  not  significantly  contribute  to 
training  transfer,  although  feedback  questionnaires  from  participants  indicated  a 
strong  desire  for  motion  feedback.  A  deeper  look  at  the  difference  between 
cabin  and  desktop  interfaces  suggested  that  the  real  contributor  may  have  been 
visual  feedback,  because  the  cabin  screens  were  larger  than  desktop  monitors. 
Interestingly,  the  school  that  served  as  the  test  site  opted  for  a  full  mission 
simulator  instead  of  the  game  approach  used  in  this  research,  despite  the 
experiment’s  indications  that  training  effectiveness  could  be  gleaned  from  this 
“serious  game.”  However,  this  study  was  somewhat  limited  in  scope.  In  this 
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study,  only  the  simulator  performance  provided  evidence  of  training 
effectiveness;  there  was  no  attempt  to  see  whether  performance  in  the  actual 
aircraft  improved  (Proctor,  Bauer,  &  Lucario,  2007). 

Other  training  transfer  experiments  explored  the  use  of  personal 
computer-based  training  outside  the  aviation  training  community.  In  2005,  two 
Naval  Postgraduate  School  students  built  a  first-person  shooter  simulation  to 
train  for  artillery  forward  observer  procedures.  Forward  Observer  Personal 
Computer  Simulation  (FOPCSIM)  allowed  a  single  user  to  learn  basic  call  for  fire 
procedures  and  practice  them  in  a  realistic  virtual  environment.  The  project  then 
took  the  simulation  to  the  Marine  Corps  basic  officer  training  course  where  call 
for  fire  is  one  of  the  fundamental  skills  taught.  They  used  FOPCSIM  to  train  one 
group  of  Lieutenants,  and  the  other  Lieutenants  were  trained  with  the  old  theater 
simulation  called  Training  Set,  Fire  Observation  (TSFO)  where  call  for  fire  results 
were  projected  in  a  slide  show.  The  research  used  the  basic  school  written  exam 
as  a  metric  for  determining  proficiency.  The  researchers  determined  that  the 
personal  computer-based  game  performed  as  well  or  better  than  the  TSFO. 
Most  notably,  FOPCSIM  was  much  more  efficient;  since  trainees  were 
automatically  scored,  they  could  perform  as  many  missions  as  their  pace  would 
allow,  instead  of  proceeding  at  the  pace  of  instructor  observation  in  TSFO. 
However,  the  research  project  was  unable  to  test  the  effects  of  the  training  in  live 
fire  (McDonough  &  Strom,  2005). 

The  training  transfer  examples  cited  so  far  focus  on  procedural  trainers  for 
individual  tasks.  This  thesis  seeks  to  explore  the  domain  of  small  unit  tactical 
operations,  and  some  research  has  investigated  the  effectiveness  of  personal 
computer  games  in  this  area.  In  2005,  a  separate  Naval  Postgraduate  School 
project  sought  to  determine  whether  commercial  off-the-shelf  (COTS)  gaming 
technology  could  improve  training.  U.S.  Army  Majors  J.  Nolan  and  J.  Jones  used 
Delta  Force:  Black  Hawk  Down-Team  Sabre™  as  a  small  unit  tactics  training 
platform  at  the  Infantry  Officer  Basic  Course  in  Fort  Benning,  GA.  The  two 
researchers  compared  survey  data  from  a  control  group  who  did  not  use 
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simulation  and  a  test  group  who  used  the  COTS  game.  They  found  only  limited 
statistical  differences  in  the  soldiers’  confidence  across  a  50-question  survey. 
Primarily,  the  surveys  indicated  that  the  simulation  training  did  not  hinder 
performance  in  any  way,  but  it  provided  virtually  no  evidence  that  the  simulation 
improved  performance.  The  researchers  described  the  most  interesting  data  as 
group  interview  results  where  the  officer  controlling  the  exercise  noticed 
improvements  in  movement  in  tactical  formations  as  well  as  better  use  of 
binoculars  in  security  tasks.  Nolan  and  Jones’s  study  demonstrated  that  serious 
games  could  be  used  for  training,  showed  no  evidence  that  the  training  technique 
was  better  than  traditional  means  (Nolan  &  Jones,  2005). 

Starting  in  2000,  the  Office  of  Naval  Research  sponsored  a  large  research 
project  called  VIRtual  Technologies  and  Environments  (VIRTE).  This  program 
studied  the  use  of  simulation  and  virtual  environments  in  many  different  ways  to 
support  the  emerging  over-the-horizon  amphibious  triad  of  the  new  landing  craft, 
amphibious  personnel  carrier,  and  tilt-rotor  aircraft.  The  project  emphasized 
training  transfer  as  the  ultimate  objective  from  the  very  beginning  (Muller,  Cohn, 
&  Nicholson,  2003).  Part  of  the  project  involved  the  development  and  evaluation 
of  a  personal  computer-based  game  called  Combined  Arms  Network  (CAN)  to 
train  Fire  Support  Teams  (FiSTs).  The  resulting  transfer  study  was  one  of  the 
most  comprehensive  personal  computer-based  game  transfer  studies  ever  done. 
Experiments  were  conducted  with  all  facets  of  the  Marine  Corps,  including  active 
and  reserve  units,  operational  units  and  schoolhouses,  basic  training  and 
advanced  training,  and  ground  and  amphibious  operations.  The  research  project 
involved  the  administration  of  surveys  and,  when  possible,  the  evaluation  of  live 
exercises  after  training.  In  general,  the  research  showed  evidence  for  training 
transfer  in  all  groups  through  increased  efficacy  ratings  on  the  surveys.  This 
project  was  unique  because  its  breadth  exposed  the  individual  appeal  of  a  given 
training  tool.  That  is,  efficacy  increased,  but  it  increased  for  different  reasons  for 
each  group  studied  (Becker  et  al.,  2009).  While  this  study  was  an  overall  training 
transfer  success,  it  points  to  the  dangers  of  over-generalizing  results. 


50 


Another  recent  transfer  study  endeavored  to  show  the  effectiveness  of 
serious  games  for  unit  training.  A  2007  study  used  a  personal  computer-based 
game  called  Close  Combat:  First  to  Fight  to  train  urban  combat  operations.  In 
this  experiment,  a  sixteen-Marine  simulation  group  used  the  game  for  training 
while  another  sixteen-Marine  group  used  traditional  training  methods. 
Performance  was  evaluated  in  a  live  fire  shoot  house,  where  the  Marines 
conducted  a  standardized  Close  Quarters  Battle  (CQB)  exercise  in  four-man 
teams.  Also,  Marines  completed  a  survey  in  which  they  rated  the  usefulness  of 
the  simulation  training  for  a  list  of  tasks  that  were  trained  in  the  exercise. 
Statistical  analysis  showed  no  difference  in  performance  in  the  live  exercise  as 
assessed  by  the  objective  grading  of  the  CQB.  However,  the  survey  results 
indicated  that  Marines  felt  the  simulation  training  increased  their  proficiency  in 
the  tactical  skills  that  were  trained  (Proctor  &  Woodman,  2007). 

6.  The  Road  Ahead 

After  looking  at  these  training  transfer  studies  as  a  group,  one  can  note 
some  important  trends: 

•  Research  has  shown  most  success  in  the  training  transfer  arena 
when  focused  on  individual,  procedural  tasks. 

•  Three  measures  of  effectiveness  are  commonly  used:  surveys  to 
gain  insight  into  participants’  self-assessed  proficiency,  knowledge 
tests  to  determine  participants’  academic  understanding  of  the 
skills;  and  instructor-  or  Subject  Matter  Expert-rated  evaluation  in 
the  real  world. 

•  Qf  the  three  measures  of  effectiveness  listed  above,  the  subjective 
self-assessed  proficiency  is  most  likely  to  show  results,  particularly 
for  collective,  cognitive  tasks  like  a  small  unit  tactical  exercise. 

•  Personal  computer-based  game  effectiveness  is  more  apparent 
when  the  research  project  involves  novices. 

•  Personal  computer-based  games  can  produce  enhanced 
performance  in  the  real  world,  but  research  has  not  yet  established 
a  pattern  or  formula  for  when  they  are  effective  and  when  they  are 
not. 
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Training  transfer  in  other  simulation  domains  has  drawn  attention  because 
of  its  difficulty,  so  researchers  have  studied  groups  of  transfer  projects  to 
determine  Best  Practices  and  pitfalls.  Boldovici  (1987)  compiled  one  of  the 
earliest  examinations  of  modern  simulation  transfer  studies.  He  listed  the 
following  pitfalls: 

•  Small  numbers  of  soldiers  or  crews  are  used  in  the  comparison; 

•  Subjects  in  the  compared  groups  are  not  matched  or  randomly 
assigned; 

•  Groups  are  treated  differently  in  respects  other  than  those  under 
investigation; 

•  Weapon  system  error  masks  training  effects; 

•  Amount  of  practice  is  insufficient  to  affect  proficiency; 

•  Ceiling  or  floor  effects  mask  differences  between  groups; 

•  Measurement  of  Task  B  performance  is  unreliable; 

•  Inappropriate  analyses  are  used  to  estimate  transfer  (Boldovici, 
1987). 

A  component  of  learning  theory  that  is  particularly  pertinent  to  the  use  of 
personal  computer-based  games  for  training  is  experiential  learning.  According 
to  this  theory,  people  learn  from  the  experience  of  the  task.  Recent  research  in 
this  topic  describes  experiential  learning  as  a  cycle  of  experiencing  the 
environment,  observing  behavior  and  reflecting  on  the  experience,  generalizing 
based  on  the  reflection,  and  modifying  concepts  based  on  new  experiences.  The 
researchers  propose  an  idea  that  is  interesting  to  military  trainers:  trainers  can 
tailor  the  environment  so  that  learners  incur  certain  experiences  according  to  a 
structured  plan  than  can  flex  to  the  needs  of  the  trainee.  In  order  to  accomplish 
such  a  plan,  the  learning  events  must: 

•  Engage  the  learner  mentally. 

•  Emulate  real-world  requirements.  Real-world  refers  to  the  physical 
environment  and  the  cognitive  tasks. 

•  Allow  the  learner  to  experience  effects  of  decisions. 
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•  Require  learner  to  reflect  on  outcomes  of  their  actions.  Build  on 
established  military  practices  of  debriefs,  lessons  learned,  and  after 
action  reports. 

•  Revisit  experiences  increasing  complexity  of  experiences  to  expand 
learners’  knowledge  and  skills  by  increasing  number  of  events, 
pacing  and  emotional  intensity  (Menaker,  Coleman,  Collins,  & 
Murawski,  2006). 

Personal  computer-based  learning  is  a  tool  for  experiential  learning,  and  the 
list  above  provides  a  formula  for  what  the  serious  game  and  the  trainer  must 
accomplish  together  in  order  for  the  trainee  to  learn.  While  this  formula  does  not 
guarantee  training  transfer  by  any  means,  it  provides  guidance  on  how  to  structure 
the  training  event  to  create  the  best  environment  for  training  transfer  to  take  place. 
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III.  PREPARATORY  EXPERIMENTS 


A.  INTERFACE  FAMILIARIZATION  PILOT  STUDY 

1.  Introduction 

Serious  games  face  a  potential  drawback  because  of  their  interfaces. 
Game  manufacturers  expend  much  effort  producing  the  most  intuitive,  efficient, 
and  user-friendly  interface  for  their  software  but,  for  any  game,  users  must  learn 
the  interface.  Military  personnel  training  with  a  serious  game  must  be  able  to 
move,  shoot,  and  communicate  as  they  would  in  live  training.  The  game 
interface  must  serve  as  an  extension  of  the  servicemember’s  warfighting  skills. 
Interface  training  clouds  the  potential  gain  of  serious  game  training  because  of 
the  extra  time  required.  If  the  individual  did  not  have  to  learn  the  game,  he  or 
she  could  do  something  more  productive.  The  military  does  not  need  gamers  for 
gaming’s  sake;  gaming  only  serves  as  a  means  to  an  end.  With  this  in  mind, 
military  serious  game  endeavors  should  strive  to  minimize  time  lost  to  learning 
the  interface. 

This  section  summarizes  a  pilot  study  conducted  as  part  of  a  project  to 
examine  the  training  effectiveness  of  Bohemia  Interactive’s  Virtual  Battlespace 
2™  {VBS  2™)  to  train  small  unit  tactics  in  a  personal  computer-based 
environment.  In  order  for  the  simulation  to  be  useful.  Marines  must  use  the 
interface  to  proficiently  drive  vehicles,  shoot  weapons,  and  maneuver  their 
bodies  as  they  would  in  the  real  world.  We  sought  to  provide  the  minimal 
computer  interface  skills  to  enable  Marines  to  operate  in  a  small  unit  mounted  or 
dismounted  environment.  We  developed  an  interface  training  program  to 
prepare  Marines  as  quickly  as  possible  so  that  they  could  devote  most  of  their 
simulation  time  to  tactics  training. 

The  military  seeks  serious  games  to  support  training  for  a  variety  of 
reasons  including  their  deployability,  relatively  low  cost,  and  flexible  training 
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environments.  Additionally,  the  military  wants  to  keep  in  touch  with  the  wired 
generation,  the  youth  who  have  experienced  a  childhood  full  of  MP3  players, 
compact  discs,  computers,  cell  phones,  and  XBoxes.  In  2008,  a  market  research 
firm  called  the  NPD  Group  reported  that  72%  of  the  U.S.  population  played  video 
games  in  2007  (Antonucci,  2008).  Later  in  the  year,  the  Pew  Internet  & 
American  Life  Project  found  that  97%  of  12-  to  17-year-old  respondents  played 
video  games,  they  played  them  often,  and  they  played  a  wide  variety  of  games 
(Irvine,  2008).  However,  the  military  cannot  count  on  gaming  skills  in  all  of  its 
personnel.  Roger  Smith  serves  as  the  Chief  Technology  Officer  for  the  U.S. 
Army  Program  Executive  Office  for  Simulation,  Training,  and  Instrumentation 
(PEO-STRI),  so,  in  looser  language,  he  is  the  Army’s  official  shopper  for  serious 
games.  In  August  2008,  Smith  responded  to  an  interview  question  about  modern 
soldiers’  familiarity  with  computer  games: 

Our  research  and  hands-on  experience  shows  that  about  50%  of 
young  enlisted  soldiers  call  themselves  “gamers”  or  are  familiar 
with  the  mechanics  of  game  play.  At  the  officer  level  it  is  around 
33%.  We  have  learned  that  we  cannot  assume  that  all  soldiers 
have  this  familiarity.  (Atkinson-Bonasio,  2008) 

A  U.S.  Army  research  effort  to  determine  the  influence  of  gaming 
experience  on  trainee  satisfaction  with  serious  games  showed  similar  results.  A 
survey  of  the  first  year  class  at  the  U.S.  Military  Academy  revealed  that  17%  of 
the  cadets  had  no  gaming  experience  and  44%  had  limited  gaming  experience. 
The  researchers  concluded  that  an  orientation  with  relevant  games  would  likely 
enhance  the  training  experience  (Orvis  et  al.,  2005).  While  the  wired  generation 
may  know  cell  phones  and  iPods,  they  do  not  necessarily  know  how  to  use  the 
games  that  support  military  training.  Moreover,  military  training  cannot  leave  the 
50%,  28%,  3%,  or  any  other  percentage  of  non-gamers  behind.  To  use  serious 
games  for  tactical  training,  all  servicemembers  must  play.  The  experience  of  the 
wired  generation  helps,  but  the  military  simulation  professional  must  craft 
exercises  with  the  novice  user  in  mind. 
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VBS  2™  has  enjoyed  some  scientific  scrutiny,  and  this  prior  research  has 
shed  some  light  on  the  time  required  to  use  the  simulation  as  an  extension  of 
one’s  warfighting  skills.  In  2004,  an  Aussie  research  team  conducted  a  weeklong 
trial  to  determine  the  potential  utility  of  VBS  2™’s  predecessor  VBS  1.  The  trials 
involved  a  group  of  participants  with  varying  degrees  of  computer  and  gaming 
experience  with  roughly  half  the  people  having  no  gaming  experience.  The  study 
determined  that  nearly  80%  of  new  users  can  attain  individual  skill  proficiency 
within  a  couple  hours.  For  higher-level  cognitive  skills  such  as  situational 
awareness  and  team  leadership,  people  need  up  to  two  days  to  become 
proficient  in  the  game  play  (Morrison  et  al.,  2005).  The  author’s  personal 
experience  with  the  simulation  at  the  I  Marine  Expeditionary  Force  (MEF) 
Simulation  Center  in  Camp  Pendleton  suggested  one  could  reduce  this  timeline. 
There,  Marines  started  training  with  only  a  30-minute  brief,  and  a  telephone  call 
confirmed  that  the  I  MEF  Simulation  Center  currently  uses  this  practice  (D. 
Gerdes,  personal  communication,  January  14,  2009).  The  I  MEF  Simulation 
Center  uses  a  locally  crafted  slide  presentation  to  brief  Marines  for  about  15 
minutes.  They  provide  about  15  minutes  of  free  practice  time  and  supply  users 
with  a  single  page  “cheat  sheet”  of  interface  commands.  Flowever,  the  I  MEF 
Simulation  Center  has  many  Marines  trained  to  help  simulation  users  during  the 
conduct  of  the  exercise.  This  pilot  study  sought  to  capitalize  on  this  model  and 
develop  an  interface  training  program  to  prepare  VBS  2™  users  within  the  30- 
minute  goal. 

As  a  pilot  study  supporting  a  larger  project,  this  project  served  two  main 
purposes.  First,  as  the  background  literature  indicates,  progress  in  future  VBS 
2™  projects  requires  knowledge  of  how  to  handle  the  interface.  One  cannot 
determine  how  well  VBS  2™  contributed  to  some  aspect  of  training  if  a  user’s 
confusion  with  the  interface  muddies  the  waters.  Thus,  this  project  started  with  a 
guess  of  an  appropriate  technique  for  accomplishing  interface  training  and 
sought  to  determine  whether  the  technique  was  adequate. 
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Second,  for  planned  future  projects,  groups  and  teams  will  use  VBS  2^^ 
as  a  common  environment  to  interact  toward  some  unit  goal.  A  lone  researcher 
in  a  laboratory  can  troubleshoot  only  a  limited  number  of  issues  in  support  of 
such  a  project.  Common  sense  dictates  that  many  unpredicted  problems  and 
quirks  will  arise  from  group  use  that  the  lone  laboratory  developer  can  never  find 
by  himself.  In  this  regard,  this  pilot  study  served  as  the  first  effort  in  the  larger 
project  of  having  multiple  people  work  together  in  a  common  environment. 

This  pilot  study  shed  light  on  the  answers  to  two  questions: 

•  Will  the  30-minute  presentation  and  practical  application  technique 
of  interface  training  suffice  to  enable  users  to  practice  small  unit 
tactical  skills  in  VBS  2™? 

•  What  unanticipated  problems  will  arise  when  a  group  operates  in  a 
common  VBS  2™  environment? 

The  pilot  study  supports  a  larger  project  to  use  VBS  2™  to  train  Marines  in 
small  unit  convoy  tactics.  That  project  involves  Marines  patrolling  a  designated 
course  in  six  to  eight  vehicles  armed  with  heavy  weapons.  The  course  trains  the 
following  skills:  react  to  an  unexploded  improvised  explosive  device  (I ED),  react 
to  an  lED  detonation,  take  immediate  action  against  a  blocked  ambush  during  a 
convoy,  take  immediate  action  against  an  unblocked  ambush  during  a  convoy, 
evacuate  a  damaged  vehicle,  and  evacuate  a  casualty.  From  these  skills,  we 
derived  a  list  of  VBS  2™  interface  skills  a  user  must  know  to  use  the  simulation 
for  the  convoy  training.  These  interface  skills  include:  individual  body 
movements,  manipulation  of  personal  gear,  use  of  personal  weapons,  vehicle 
interaction  and  operation,  use  of  vehicle  weapons,  manipulating  a  casualty, 
towing  a  vehicle,  disarming  an  lED,  and  recognition  of  friendly,  civilian,  and 
enemy  avatars.  The  pilot  study  endeavored  to  train  these  interface  skills. 

To  answer  the  first  question  listed  above,  we  developed  a  criterion  for 
success.  We  based  success  on  the  level  of  comfort  users  felt  in  performing  each 
of  the  tactical  skills  listed  above.  We  determined  this  comfort  level  through  Likert 
scaled  responses  to  each  of  the  skills.  We  viewed  success  as  a  mid-level 
comfort  rating  for  all  skills. 
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2. 


Method 


a.  Participants 

Twelve  Naval  Postgraduate  School  Modeling,  Virtual  Environment, 
and  Simulation  (MOVES)  graduate  students,  ranging  in  age  from  27  to  41  years, 
participated  in  the  pilot  study.  All  participants  volunteered  as  part  of  a  seminar 
course  on  current  simulation  technology.  All  the  participants  were  male  military 
officers  with  service  times  ranging  from  4  to  24  years.  Four  foreign  officers 
participated,  and  U.S.  officers  represented  all  four  of  the  armed  services. 

b.  Apparatus 

The  project  used  twelve  Dell  Precision  M6300  laptops  from  a  suite 
of  the  Marine  Corps  Deployable  Virtual  Training  Environment  (DVTE)  package. 
Peripheral  equipment,  including  mice,  cables,  and  switches,  came  from  the 
standard  DVTE  package.  For  the  familiarization  training  and  evaluation,  we 
networked  the  computers  in  pairs  with  four  computers  per  switch.  As  shown  in 
Figure  2,  paired  users  faced  each  other  at  a  classroom  table  so  that  they  could 
not  see  each  other’s  screens.  For  the  follow-up  evaluation,  all  twelve  computers 
were  networked  together  in  the  same  environment  using  two  switches  with  six 
computers  per  switch.  Participants  operated  as  individuals  in  the  follow-up 
evaluation. 
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Figure  2.  Classroom  setup  for  12  participants 

c.  Procedure 

This  project  used  the  I  MEF  Simulation  Center’s  training  model  of  a 
slide  presentation  with  practice  time.  The  project  also  provided  a  single  page 
cheat  sheet  of  interface  commands,  included  in  Appendix  E.  The  project 
modified  the  interface  training  by  having  the  trainees  follow  along  in  the 
simulation  on  their  own  DVTE  computers  during  the  slide  presentation.  The  I 
MEF  Simulation  Center  cannot  use  this  method  because  of  the  physical  layout  of 
the  facility,  but  we  assumed  that  the  hands-on  application  would  enhance  the 
interface  training.  With  this  construct  in  mind,  the  project  developed  a 
familiarization  scenario  to  guide  novice  users  through  the  basic  individual, 
weapon,  and  vehicle  skills  necessary  to  use  VBS  2™. 
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To  support  the  development  of  this  scenario,  the  project  started  by 
identifying  the  specific  interface  skills  needed  for  future  tactical  training.  The  pilot 
study  supports  a  larger  project  oriented  on  tactical  convoy  training.  The  Marine 
Corps  convoy  training  supports  well-documented  service  training  objectives,  and 
analysis  of  these  training  objectives  clarified  the  interface  skills  needed  for  the 
pilot  study.  An  individual  must  be  able  to  maneuver  his  body  dismounted, 
including  walking,  running,  and  getting  into  and  out  of  the  prone  position.  He 
must  be  able  to  use  personal  gear  available  in  the  simulation,  including  compass, 
binoculars,  global  positioning  system  (GPS),  watch,  and  night  vision  goggles 
(NVGs).  The  user  must  be  able  to  use  personal  weapons,  including  rifle  and 
grenade,  to  accurately  engage  targets.  Other  individual  skills  include  recognition 
of  civilians  and  enemy  combatants,  recognition  of  improvised  explosive  devices 
(lEDs),  and  casualty  evacuation.  In  convoy  operations.  Marines  must  be  able  to 
operate  vehicles  and  vehicle  weapons  proficiently.  Specialized  vehicle 
operations  necessary  for  the  convoy  operation  include  towing  a  damaged 
vehicle. 

Typically,  Marines  will  operate  in  a  convoy  scenario  in  teams  of 
three;  a  vehicle  commander  riding  in  the  passenger  seat,  a  driver,  and  a  gunner 
for  the  vehicle’s  weapon  system.  The  vehicle  commander  does  not  have  to  learn 
any  specific  vehicle  interface  skills  like  the  driver  or  gunner,  but  he  must  be 
familiar  with  both.  For  this  reason,  the  interface  training  was  developed  for 
teams  of  two,  specialized  for  the  driver  and  the  gunner.  Vehicle  commanders 
could  be  trained  using  either  station. 

The  interface  training  scenario  involved  two  parts:  individual 
training  and  vehicle  training.  For  the  individual  training,  two  Marines  operated  in 
the  same  environment  that  included  a  single  personal  weapons  range.  Users 
started  in  two  separate  lanes  that  mirrored  each  other  with  the  range  in  the 
middle.  Each  user  started  play  in  front  of  his  own  vehicle.  At  this  position,  the 
user  could  learn  individual  movements.  Both  users  moved  to  firing  points  on  the 
personal  weapons  range  and  practiced  M16A4  rifle  and  grenade  target 
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engagement.  Also,  users  practiced  using  their  personal  gear.  After  completing 
these  exercises,  users  returned  to  their  individual  vehicles  to  start  the  vehicle 
training. 

Users  started  the  vehicle  training  by  driving  high  mobility  multi¬ 
purpose  wheeled  vehicles  (HMMWVs)  along  a  designated  course  to  a  vehicle 
weapons  range.  There,  they  used  the  vehicle’s  0.50  caliber  machine  gun  to 
practice  engaging  vehicle  targets.  After  practicing  with  the  vehicle’s  weapon 
system,  the  users  followed  a  designated  course  to  a  middle  Eastern-style  house 
where  they  could  get  out  of  the  vehicle  and  examine  each  other,  three  civilians, 
and  two  enemy  insurgents.  Each  user  shot  an  insurgent  and  practiced  putting 
the  body  in  a  vehicle.  Users  also  viewed  an  lED  and  practiced  disarming  it. 
After  personnel  and  lED  recognition,  the  users  got  in  a  7-ton  truck,  the  medium 
tactical  vehicle  replacement  (MTVR),  and  followed  a  designated  course  to  a 
damaged  Landrover.  There,  they  practiced  towing  the  vehicle.  After  completion 
of  the  vehicle  towing,  the  users  had  five  to  ten  minutes  of  free  practice  time.  The 
entire  familiarization  course,  shown  in  Figure  3,  occupied  a  3  by  4  kilometer  area 
of  the  Cleghorn  region  of  the  Twentynine  Palms  terrain  database  available  in 
VBS  2™. 
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We  designed  an  evaluation  course  in  the  familiarization  scenario  to 
determine  the  interface  skill  level  achieved  by  the  users  (Figure  4).  Users  started 
in  a  fresh  MTVR  at  a  known  command  post  location  and  used  signs  to  follow  a 
route  on  a  mission  to  tow  a  damaged  HMMWV.  Along  the  route,  they 
encountered  four  stations  briefed  immediately  prior  to  the  exercise.  Station  1 
included  three  enemy  trucks,  Station  2  included  a  four-man  enemy  fire  team. 
Station  3  included  a  compound  with  two  enemy  combatants  inside  and  a  well- 
marked  lED  outside,  and  Station  4  included  another  four-man  enemy  fire  team. 
A  properly  executed  exercise  would  result  in  the  three  trucks  destroyed,  all  ten 
personnel  targets  killed,  one  of  the  enemy  from  the  compound  loaded  into  the 
truck,  the  lED  disarmed,  and  the  damaged  HMMWV  in  tow  with  the  MTVR  on  the 


Figure  3.  Familiarization  training  area 
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nearest  road  ready  to  return.  All  users  had  the  following  resources:  a  map  of  the 
area,  a  diagram  of  the  familiarization  training  area,  and  a  cheat  sheet  of  interface 
commands. 
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Figure  4.  Team  evaluation  course 


In  order  to  assess  the  degradation  of  interface  skills  over  time,  the 
project  included  a  follow-up  evaluation  one  week  later.  The  exercise  had  users 
shoot  an  enemy  insurgent,  drive  a  vehicle  to  the  body,  and  load  the  body  into  the 
vehicle.  They  destroyed  an  enemy  truck  with  a  grenade.  Then  the  users 
assumed  a  prone  position,  shot  a  target  less  than  100  meters,  and  shot  a  target 
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farther  than  100  meters.  Users  followed  the  instruction  sheet  included  in 
Appendix  I  to  keep  track  of  each  task  accomplished.  All  participants  used 
identical  courses  (in  design,  a  prototype  course  was  copied  and  pasted  to  make 
the  others)  about  200  meters  in  length  in  VBS  2™’s  Samawah  terrain  database 
(Figure  5).  All  users  had  an  instruction  sheet  that  explicitly  listed  each  step  of  the 
exercise.  Half  of  the  users  had  the  cheat  sheet  of  interface  commands  from  the 
week  before;  the  other  half  had  no  interface  assistance. 


Figure  5.  Individual  evaluation  course 


VBS  2™  offers  a  robust  after  action  review  (AAR)  tool  for  debriefing 
training  exercises.  This  tool  allows  the  trainer  to  replay  a  recorded  exercise 
using  a  viewing  camera  that  can  be  positioned  anywhere  in  the  scene.  The 
trainer  can  attach  the  camera  to  an  object,  such  as  a  vehicle,  or  fly  the  camera 
through  the  scene  to  any  point.  The  trainer  can  start,  stop,  fast  forward,  and 
rewind  the  AAR  as  required.  In  this  way,  a  trainer  can  review  any  part  of  a 
tactical  scenario  multiple  times  from  multiple  positions.  The  AAR  tool  includes  a 
timer  enabling  the  recording  of  specific  event  durations.  The  project  used  this 
AAR  tool  for  data  collection  in  support  of  the  evaluation. 

Participants  completed  a  one-page  pre-exercise  survey  that 
included  demographic  information  and  computer  usage  profile.  They  chose  one 
of  the  twelve  simulation  computers  available  in  an  open  classroom.  After  a  brief 
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project  introduction,  the  participants  saw  the  slide  presentation  and  completed 
the  individual  training  portion  of  the  exercise.  They  were  then  informed  that  they 
were  operating  as  teams  of  two  and  started  the  vehicle  familiarization  training. 
After  the  personnel  and  lED  recognition,  the  participants  boarded  the  same 
vehicle  and  completed  the  rest  of  the  training  together.  They  completed  the 
towing  training  and  free  practice  as  described  above,  proceeded  to  the  command 
post,  and  boarded  the  evaluation  MTVR.  Participants  had  ten  minutes  to 
complete  the  evaluation  exercise,  recorded  with  the  AAR  tool.  After  the  exercise 
finished,  participants  completed  a  one-page  post-exercise  questionnaire  that 
focused  on  skills  that  were  easy  and  difficult,  memory  of  interface  skills,  and  a 
seven-question  Likert  scaled  evaluation  of  confidence  in  the  basic  interface  skills. 
The  responses  to  the  basic  skills  involved  a  rating  from  1  to  5,  and  we 
considered  a  rating  of  3  as  average  for  determining  the  success  of  the  training. 
The  two  surveys  used  for  the  experiment  are  included  in  Appendix  H.  The 
complete  exercise,  including  training  and  evaluation,  lasted  for  a  50-minute  class 
period.  The  follow-up  evaluation  occurred  one  week  later,  in  the  first  five  minutes 
of  the  class  period. 

3.  Results 

First  and  foremost,  a  pilot  study  serves  to  identify  unforeseen  problems  in 
the  project  methodology.  This  pilot  study  accomplished  this  goal  with  the 
following  observations.  Because  of  a  technical  software  issue,  disarming  an  I  ED 
crashed  the  AAR  tool.  For  this  reason,  all  participant  activity  recorded  after  lED 
disarming  was  lost  to  analysis.  The  AAR  tool  shows  shots  fired  on  a  time  bar,  so 
one  could  determine  whether  the  team  fired  after  the  I  ED  crashed  the  AAR  tool 
even  though  the  lED  disarming  prohibited  viewing  this  portion  of  the  scenario. 
Lack  of  clarity  in  the  instructions  resulted  in  participants  driving  their  FIMMWVs 
instead  of  the  MTVR  to  the  Landrover  for  the  towing  training.  The  HMMWVs 
could  not  tow,  so  the  participants  did  not  train  that  skill,  but  had  a  verbal 
explanation  of  the  towing  procedure.  Participants  tended  to  assault  the  insurgent 
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fire  teams.  Although  the  enemy  objects  were  set  to  “never  fire,”  they  fired  to 
defend  themselves  when  in  imminent  danger,  resulting  in  unexpected  close- 
range  small  arms  battles.  Finally,  in  the  second  session  evaluation,  one  of  the 
participants  shot  another  right  at  the  beginning  of  the  scenario,  so  that  the  victim 
participant  was  removed  from  all  data  collection.  These  unexpected  problems 
confounded  the  data  collected  from  the  evaluation  exercise. 

a.  Team  Evaluation 

Three  of  the  six  teams  completed  the  full  evaluation  with  a  mean 
time  of  7:11  (standard  deviation  1:32).  Four  of  the  six  teams  completed  the 
evaluation  through  the  I  ED  station  with  a  mean  time  of  6:54  (standard  deviation 
1:30).  All  six  teams  engaged  and  destroyed  the  three  trucks  at  the  first  station. 
Four  of  the  six  teams  assaulted  Fire  Team  1  by  driving  right  up  to  the  enemy.  As 
previously  described,  the  artificial  intelligence  allowed  the  enemy  to  defend  itself 
when  in  imminent  danger,  resulting  in  close-quarters  battles  with  Fire  Team  1. 
For  this  reason,  one  team  lost  both  members  at  the  Fire  Team  1  station  and 
could  not  continue.  Two  teams  continued  the  scenario  with  one  member  dead;  in 
both  cases,  the  single  team  member  completed  the  scenario  alone. 

All  six  teams  eventually  followed  the  correct  path  as  designated  by 
the  signs.  However,  four  of  the  six  teams  showed  signs  of  disorientation  at  some 
point  during  the  exercise.  Two  of  the  teams  strayed  so  far  from  the  path  that 
they  clearly  relied  on  their  maps  to  navigate  back. 

As  mentioned,  three  teams  did  not  complete  the  course.  One  team 
did  not  complete  the  course  because  Fire  Team  1  killed  both  members.  For  the 
team  that  completed  the  course  through  the  lED  station,  but  did  not  complete  the 
course  entirely,  one  of  the  team  members  faced  a  unique  problem.  The  team 
member  was  in  the  prone  position  and  remained  that  way.  The  member  could 
move  forward  by  low  crawling,  but  could  not  stand.  This  behavior  indicated  that 
the  team  member  had  become  injured,  although  it  was  unclear  how  this  had 
happened.  The  member’s  inability  to  stand  made  disarming  the  lED  impossible. 
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The  third  team  chose  to  disarm  the  lED  at  the  compound  before  engaging  the 
enemy  inside  and  evacuating  the  casualty.  For  this  reason,  the  AAR  tool  did  not 
capture  any  further  action  at  the  compound.  However,  we  could  determine  that 
the  team  fired  on  Fire  Team  2  later  in  the  scenario. 

b.  Survey  Results 

The  post-exercise  survey  provided  some  insight  into  how 
participants  felt  about  their  training.  Participants  answered  two  knowledge 
questions  on  the  post-exercise  survey.  Without  the  aid  of  the  simulation  or  the 
cheat  sheet,  all  participants  correctly  knew  how  to  move  their  body  forward. 
However,  only  five  of  the  twelve  participants  correctly  explained  how  to  tow  a 
vehicle.  Of  these  five  correct  respondents,  four  were  drivers. 

Participants  responded  to  two  subjective  questions  about  the  skill 
or  skills  they  found  most  difficult  and  the  skill  or  skills  they  found  easiest.  Nine  of 
the  twelve  participants  found  shooting  easiest,  and  two  found  body  movements 
easiest.  Answers  to  the  question  about  the  most  difficult  task  differed 
considerably  and  included  the  following:  towing,  disarming  an  lED,  driving 
backward,  navigation,  confirming  targets,  unslinging  a  weapon,  and  recovering  a 
casualty.  A  group  interview  question  immediately  following  the  exercise  revealed 
that  reported  driving  problems  were  associated  with  towing,  because  the  driver 
must  maneuver  around  the  damaged  vehicle  and  back  up  to  it.  The  driver  has 
no  mirrors  or  other  means  to  see  behind  him,  so  another  individual  in  the 
simulation  must  guide  him. 

Participants  responded  to  a  question  about  their  comfort  level  with 
each  of  the  interface  skills  on  a  five-point  Likert  scale,  with  large  numbers 
reflecting  the  highest  degree  of  confidence  in  that  skill.  Figure  6  shows  the 
results  for  the  following  six  skills:  basic  body  movement,  vehicle  maneuvering, 
shooting  individual  weapons,  shooting  vehicle  heavy  weapons,  manipulating  a 
casualty,  disarming  an  lED,  and  towing  a  vehicle.  Participants  felt  least  confident 
with  the  casualty  evacuation  and  towing  tasks.  Review  of  the  AAR  tool 
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recordings  confirms  that  these  tasks  were  more  difficult  than  shooting  and 
moving.  Confidence  ratings  exceeded  an  average  of  3.0  for  all  tasks  except 
towing. 
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Figure  6.  Subjective  ratings  of  confidence  in  interface  skills 

c.  Follow-on  Individual  Evaluation 

The  follow-on  evaluation  conducted  one  week  later  demonstrated 
considerable  degradation  of  previously  taught  interface  skills.  Only  eleven 
participants  conducted  the  exercise,  because  one  participant  shot  another 
immediately  at  the  beginning  of  the  exercise,  eliminating  the  victim  from  further 
play.  Only  two  of  the  eleven  remaining  participants  completed  the  exercise 
correctly.  An  additional  participant  completed  all  of  the  events  except  standing 
and  recording  his  watch  time.  He  accomplished  this  in  less  time  than  the  two 
who  completed  the  exercise.  This  participant  was  a  foreign  officer  and  might  not 
have  understood  the  instructions  fully  to  complete  the  last  two  events.  Ten  of  the 
eleven  participants  shot  the  first  target  (the  eleventh  successfully  shot  a  target, 

69 


but  it  was  another  participant).  All  eleven  participants  successfully  interacted 
with  the  HMMWV.  Seven  of  the  eleven  participants  successfully  loaded  the 
casualty  into  the  HMMWV.  Only  four  participants  threw  a  grenade  at  the  truck 
target,  and  those  four  participants  also  successfully  engaged  the  short  and  far 
targets.  Five  of  the  eleven  participants  successfully  used  their  simulation  watch 
to  record  a  time. 

During  the  follow-on  individual  evaluation,  half  of  the  participants 
had  the  cheat  sheet  from  the  previous  week’s  training,  and  half  did  not.  Of  the 
four  participants  who  successfully  completed  the  exercise  through  shooting  the 
far  target,  two  had  cheat  sheets  and  two  did  not.  Both  the  participant  who  shot 
another  participant  and  his  victim  had  cheat  sheets.  The  four  participants  who 
shot  the  far  target  had  times  of  1 :54,  2:23,  2:40,  and  5:08. 

The  follow-on  individual  evaluation  employed  robust  measures  to 
guide  participants  through  the  exercise.  Participants  had  a  sheet  with  all 
instructions  explicitly  listed.  The  course  was  simple;  no  other  objects  cluttered 
the  terrain  except  the  targets  for  the  scenario.  Cones  provided  a  maneuver  box 
for  each  participant.  Despite  the  clear  verbal  and  written  instructions  and  visual 
cues  in  the  simulation,  one  participant  still  managed  to  get  disoriented  and 
confused.  Not  only  did  this  participant  kill  another,  he  drove  across  another 
participant’s  area  and  loaded  the  victim  in  his  HMMWV. 

d.  Practical  Tips  for  Foiiow-on  Work 

In  addition  to  determining  whether  the  proposed  interface  training 
sufficed,  the  pilot  study  provided  an  opportunity  to  investigate  practical  aspects  of 
conducting  research  with  VBS  2™.  The  Marine  Corps  intends  DVTE,  including 
VBS  2™,  to  be  portable.  The  pilot  study  site  was  set  up  for  the  single  class 
period  of  the  exercise  only.  We  set  up  the  system  twice  in  this  project:  once  for 
the  training  session  and  once  for  the  follow-up  evaluation.  We  conducted  an 
equipment  rehearsal  prior  to  the  first  session  of  the  pilot  study,  which  provided 
several  lessons  learned.  Based  on  the  equipment  rehearsal,  we  diagrammed  the 
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pilot  study  site  with  the  intent  to  set  up  all  twelve  computers,  network  them,  and 
run  the  scenario  in  a  50-minute  period.  Our  setup  script  and  diagram  that 
resulted  from  the  equipment  rehearsal  are  included  in  Appendix  G.  We  found 
that  we  had  all  hardware  set  up  and  networked  within  25  minutes  for  both 
sessions.  Timing  for  starting  scenarios  varies  depending  on  the  terrain  database 
that  is  loaded  and  the  networking  arrangement.  For  example,  the  Twentynine 
Palms  database  requires  much  more  processing  power  than  any  of  the  other 
installed  databases.  Because  the  first  session  involved  six  separate 
environments  (one  for  each  pair  of  participants)  and  the  Twentynine  Palms 
database,  we  took  15  minutes  to  get  all  computers  running.  With  a  single 
environment  and  the  Samawah  database  for  the  second  scenario,  we  only 
required  five  minutes  to  start  the  scenario.  In  both  sessions,  with  the  help  of  the 
twelve  participants,  all  twelve  computers  were  shut  down  and  stowed  within  ten 
minutes.  This  timing  information  is  critical  for  future  research  efforts  that  will 
involve  setting  up  a  group  of  computers  and  stowing  them  after  the  experiment. 

During  the  team  evaluation  exercise,  one  participant  experienced 
mild  motion  sickness  and  had  to  stop  participation  in  the  exercise  for  a  short 
time.  The  participant  filled  the  role  of  driver.  He  reported  that  he  had 
experienced  motion  sickness  in  simulators  before,  and  he  had  used  simulations 
in  professional  training. 

4.  Discussion 

This  pilot  study  sought  to  develop  a  training  program  that  could  prepare 
novice  users  to  properly  and  efficiently  use  the  VBS  2™  interface  to  conduct 
tactical  convoy  operations  within  30  minutes.  For  our  survey  success  criterion, 
the  training  resulted  in  average  or  above  average  comfort  levels  for  all  skills 
except  towing.  Because  of  the  technical  problems  in  teaching  the  towing  task, 
we  eliminated  towing  from  the  criteria  and  concluded  that  the  training  met  the 
success  criterion.  While  the  pilot  study  highlighted  some  concerns  that  must  be 
addressed  for  future  work,  the  results  of  the  project  confirmed  that  the  scenario 
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and  training  construct  sufficed  to  familiarize  users  with  the  interface.  The  30- 
minute  training  period  can  be  used  for  timing  planning  for  future  research. 

The  30-minute  training  period  most  successfully  taught  the  individual  skills 
of  moving  and  shooting,  although  participants  had  difficulty  cycling  between 
individual  weapons  in  the  individual  evaluation.  The  more  complex  skills  of 
casualty  evacuation,  towing,  and  disarming  an  lED  caused  the  most  problems. 
However,  when  the  participants  worked  in  teams  of  two,  they  generally  figured 
out  how  to  deal  with  these  tasks  as  a  team.  As  a  team,  participants  experienced 
more  trouble  with  navigation  and  surviving  a  threatened  enemy  in  close  quarters. 

Observation  of  the  trainees  throughout  the  group  and  individual 
evaluations  indicated  that  participants  had  sufficient  training  but  needed  more 
practice.  The  explanation  and  demonstration  sufficed  to  give  participants  a  basic 
understanding  of  how  to  use  the  interface.  The  cheat  sheet  provided  a  sufficient 
resource  to  fill  in  gaps  in  knowledge  for  those  tasks  not  exercised  regularly. 
Participants  simply  needed  to  practice  more  to  learn  the  interface  and  the  cheat 
sheet.  Thus,  the  success  of  the  interface  training  must  be  measured  relative  to 
what  will  succeed  it.  Participants  do  not  complete  the  interface  training  ready  to 
be  tested  in  a  larger  exercise  using  VBS  2™  as  an  extension  of  their  warfighting 
skills.  However,  the  interface  training  prepares  them  to  start  working  on  more 
complex  tactical  tasks  that  would  inherently  provide  the  interface  practice  needed 
for  larger  exercises.  Estimating  the  amount  of  practice  necessary  to  use  the 
interface  proficiently  is  beyond  the  scope  of  this  project,  but  this  study  indicated 
that  the  Aussie  research  team  cited  in  this  thesis’s  background  provides  a  correct 
estimation  of  two  hours  or  less  for  the  average  user. 

The  errant  participant  in  the  follow-on  individual  evaluation  highlighted  the 
issue  of  fratricide.  Despite  the  care  taken  to  expose  participants  to  the  different 
types  of  avatars  in  VBS  2™,  one  participant  did  not  correctly  distinguish  friendly 
targets  at  close  range.  This  incident  suggests  further  emphasis  on  the  avatar 
recognition  portion  of  the  training. 
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For  future  repetitions  of  this  experiment,  some  means  of  minimizing  the 
confusion  experienced  by  the  errant  participant  must  be  devised.  Based  on  this 
project,  two  potential  solutions  suggested  themselves.  First,  the  training  scenario 
used  a  waypoint  at  the  beginning  to  guide  users  to  the  shooting  range.  The 
waypoint  provides  an  arrow  in  the  user’s  screen,  indicating  the  direction  he 
should  go.  The  waypoint  also  provides  a  distance  to  travel.  Waypoints  should 
be  used  to  help  the  participant  find  his  way.  Second,  the  AAR  function  could 
potentially  help  participants.  The  researcher  could  conduct  the  follow-on 
individual  exercise  himself,  using  the  AAR  to  record  an  example  of  a  correct 
execution.  The  researcher  could  then  play  this  AAR  prior  to  the  follow-on 
individual  evaluation  to  provide  the  user  a  visual  demonstration  of  how  the 
exercise  should  be  done.  From  that  point,  the  participant’s  only  task  would  be  to 
manipulate  the  interface  to  repeat  what  he  had  just  seen.  This  visual  display 
may  help  the  user  avoid  the  confusion  of  the  scenario,  thereby  focusing  on  the 
interface  tasks. 

The  unintended  problem  in  the  towing  training,  in  which  participants 
moved  to  the  Landrover  in  the  wrong  vehicles,  provided  a  valuable  comparison 
study.  The  towing  operation  involves  steps  roughly  comparable  to  loading  a 
casualty  into  a  vehicle.  Of  all  the  interface  skills,  participants  were  most 
uncomfortable  with  towing.  Because  most  participants  achieved  proficiency  in 
skills  of  similar  complexity,  the  value  of  hands-on  practice  during  the  training 
became  clear.  Having  participants  follow  along  in  their  own  simulation 
environments  during  the  brief  appeared  to  produce  better  results  than  a  verbal 
explanation  of  the  procedures. 

The  empirical  data  from  the  study  did  not  demonstrate  that  participants 
depended  on  the  cheat  sheet.  Nevertheless,  participants  conducted  an  exercise 
for  fun  after  the  follow-on  individual  evaluation.  All  of  the  participants  without  the 
cheat  sheet  were  grateful  to  get  it  back  for  the  next  exercise.  Participants  were 
observed  consulting  the  cheat  sheet  during  the  exercise. 
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A  pilot  study  primarily  provides  lessons  learned  for  future  work.  This  study 
highlighted  some  issues  that  could  endanger  a  future  research  effort  if  not 
corrected.  Specific  issues  identified  in  this  pilot  study  include: 

•  Ensure  that  lED  disarmament  does  not  interfere  with  AAR 
recording. 

•  Brief  participants  to  remain  on  the  road,  so  they  do  not  provoke  the 
enemy  targets  to  engage  them  in  close-quarters  battle. 
Alternatively,  make  the  enemy  targets  impotent  by  reducing  their 
ammunition  levels  to  zero. 

•  In  the  towing  training,  clarify  vehicle  mounting  so  that  participants 
use  the  MTVR. 

•  Screen  for  previous  simulator  sickness,  and  prepare  training  units 
for  the  fact  that  some  participants  may  not  be  able  to  handle  the 
simulator. 

•  Brief  rules  of  engagement  for  the  evaluation  exercise  (that  is, 
engage  all  targets  upon  identification). 

•  Emphasize  the  key  control  to  raise  and  lower  the  rifle. 

•  On  the  survey,  the  computer  use  question  implies  the  individual’s 
personally  owned  computer.  Divide  the  question  into  personally 
owned  computer  use,  and  government  and  other  computer  use,  to 
encompass  all  of  a  participant’s  computer  time. 

•  On  the  survey,  make  a  block  for  the  computer  number  clearly  at  the 
top  of  the  page  for  clarity. 

•  VBS  2™  offers  two  modes  of  operation:  user  and  administrator. 
Connect  participants  as  users  instead  of  administrators.  One 
participant  was  able  to  learn  how  to  use  administrator  rights  to 
interfere  with  the  conduct  of  the  exercise.  Connecting  the 
participants  as  users  also  allows  the  assignment  of  unique  names, 
which  would  make  data  collection  much  simpler. 

Some  notable  features  of  the  pilot  study  that  worked  well  deserve 
consideration  in  future  work: 

•  The  general  construct  of  the  presentation  combined  with  hands-on 
practice  worked  well  for  all  participants.  The  familiarity  training 
scenario  was  constructed  well  to  support  the  training.  The  cheat 
sheet  worked  well  as  written. 

•  The  equipment  rehearsal  provided  valuable  lessons  learned.  Had 
this  not  been  done,  a  number  of  potential  pitfalls  could  have 
jeopardized  the  pilot  study  before  it  even  started. 
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•  Two  resources  supported  the  researchers  setting  up  the  pilot  study 
setting.  A  script  of  all  the  tasks  that  needed  to  be  accomplished 
provided  explicit  instructions  for  setup.  A  diagram  of  the  computers 
and  wiring  connections  accompanied  the  script.  The  assistant 
never  consulted  the  text  of  the  script,  but  relied  heavily  on  the 
diagram.  Future  VBS  2™  projects  will  involve  various  assistants. 
More  effort  should  be  put  into  wiring  schematics,  and  the  script 
should  not  be  provided  at  all. 

•  For  the  researcher  conducting  the  study,  a  script  was  absolutely 
necessary.  The  interface  training,  group  evaluation,  and  follow-on 
evaluation,  were  scripted  in  detail  and  well  rehearsed.  The  script 
and  rehearsal  were  valuable  tools  to  minimize  critical  errors  in  the 
conduct  of  the  study. 

In  summary,  use  of  VBS  2™  as  a  tactical  training  tool  depends  heavily  on 
interface  training.  Without  adequate  interface  training  and  experience.  Marines 
will  not  be  able  to  use  the  simulation  as  an  extension  of  their  warfighting  skills, 
resulting  in  unintended  problems.  While  a  30-minute  training  session  does  not 
make  a  novice  into  a  proficient  VBS  2™  user,  the  training  suffices  to  get  the 
person  started  so  that  the  individual  only  needs  to  practice  to  achieve  interface 
proficiency.  Interface  training  works  well  if  trainees  can  work  hands-on  in  the 
simulation  during  presentation.  A  brief  cheat  sheet  does  not  make  a  user 
proficient  by  itself,  but  provides  support  when  needed.  This  study  provided 
several  practical  insights  for  future  work.  Key  improvements  involved  technical 
issues  with  the  AAR  tool  and  connecting  computers  in  user  mode.  Key 
successes  included  the  value  of  preparation  and  rehearsal  and  the  importance  of 
an  exercise  script.  Finally,  the  study  provided  hints  into  how  confused  and 
disoriented  a  trainee  may  become  in  even  the  simplest  of  VBS  2™  simulations — 
a  consideration  that  will  certainly  impact  future  work. 

B.  SCENARIO  DEVELOPMENT  PILOT  STUDY 


1.  Introduction 

This  section  summarizes  a  second  pilot  study  conducted  as  part  of  a 
project  to  examine  the  training  effectiveness  of  VBS  2™  to  train  small  unit  tactics 
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in  a  personal  computer-based  environment.  This  project  involved  the  testing  of  a 
specific  small  unit  tactical  convoy  scenario.  In  evaluating  the  scenario,  we 
sought  to  determine  whether  the  training  improved  the  participants’  appreciation 
of  serious  games  as  a  training  tool.  Furthermore,  we  investigated  whether 
participants  felt  that  their  small  unit  tactical  convoy  knowledge  and  skills 
improved  as  a  result  of  the  training.  These  questions  served  as  a  first  step 
toward  determining  whether  VBS  2™  can  provide  effective  tactical  training  for 
military  personnel. 


a.  The  Training  Exercise  That  Started  it  Aii 

At  the  Marine  Corps  Air  Ground  Combat  Center  (MCAGCC)  in 
Twentynine  Palms,  the  Tactical  Training  Exercise  Control  Group  (TTECG) 
administers  a  series  of  training  scenarios  to  operational  units.  The  TTECG  has 
developed  these  scenarios  over  time  against  the  backbone  of  the  training  and 
readiness  system.  As  a  starting  point  for  our  scenario  development,  we 
observed  TTECG  operations.  TTECG  supports  a  series  of  exercises  packaged 
under  the  name  Mojave  Viper.  Marine  Corps  fleet  units,  both  active  and  reserve, 
rotate  through  Mojave  Viper  training  at  the  rate  of  twelve  units  per  year.  TTECG 
exercises  are  rigidly  controlled  and  evaluated  in  the  effort  to  provide  all  units  the 
same  training  experience  and  a  common  evaluation  baseline.  The  Motorized 
Operations  Course  (MOC)  provides  a  tactical  scenario  with  four  individual 
stations  to  test  the  standard  operating  procedures,  immediate  action  drills,  and 
small-unit  tactics  of  a  platoon-sized  convoy  (usually  between  six  and  eight 
vehicles). 

The  first  station  tests  the  unit’s  ability  to  recognize  an  I  ED,  secure 
the  area,  report  the  appropriate  information  to  higher  headquarters,  and  link  up 
with  forces  who  will  handle  the  device.  The  station  consists  of  a  vehicle  hulk 
representing  a  vehicle  with  an  lED  inside.  The  second  station  requires  a  unit  to 
react  to  an  exploding  lED  and  a  far  ambush.  The  exercise  controller  causes  an 
explosion  (one-pound  stick  of  TNT)  and  informs  the  unit  that  the  simulated  bomb 
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disabled  the  lead  vehicle.  The  controller  then  activates  ten  personnel  targets  at 
long  range  (350  meters).  The  participating  unit  can  choose  to  suppress  the 
targets  and  maneuver  the  unit  through  or  suppress  the  targets  and  assault.  The 
third  station  consists  of  a  destroyed  enemy  battle  position  with  a  near  ambush 
consisting  of  trenches  with  bunkers  and  personnel  targets.  Again,  the  unit  may 
choose  to  suppress  and  maneuver  through  or  assault  the  targets.  Regardless  of 
the  drill  employed,  the  controller  will  assess  one  or  more  casualties  for  the  unit  to 
handle.  If  supporting  aviation  assets  are  available,  the  controller  will  paint  enemy 
units  to  the  north.  Station  4  allows  the  unit  to  exercise  control  of  tactical  aircraft 
to  destroy  two  vehicles. 

The  MOC  provides  the  opportunity  for  the  exercise  unit  to  practice 
several  distinct  skills.  The  unit  must  execute  “fire  and  maneuver,”  the  set  of  skills 
used  to  direct  and  control  multiple  vehicles  to  destroy  targets  while  maintaining 
unit  security.  All  vehicles  must  practice  fire  discipline  so  that  all  targets  get 
destroyed  without  expending  all  ammunition  on  one  target  and  letting  others  go 
free.  Internal  communications  must  be  correct  and  effective.  The  unit  must 
exercise  control  over  intelligence  and  combat  air  support  assets  in  a  tactically 
meaningful  way.  The  unit  must  handle  battlefield  realities  such  as  casualties  and 
vehicle  breakdown.  The  unit  must  coordinate  its  actions  with  higher 
headquarters,  reporting  all  significant  activity  as  appropriate.  The  exercise  tests 
the  unit’s  understanding  and  practical  application  of  convoy  tactics.  The  exercise 
unit  demonstrates  the  ability  to  conduct  the  various  immediate  action  drills 
required  for  tactical  success.  Most  importantly,  the  MOC  provides  an  opportunity 
to  scrutinize  a  unit’s  standard  operating  procedures  to  determine  whether  they 
are  really  standard,  and  whether  they  are  truly  understood. 

In  Chapter  II,  we  discussed  NAVMC  3500.87,  the  Infantry  Training 
and  Readiness  (T  &  R)  Manual,  that  serves  as  the  basis  of  all  infantry  training. 
The  publication  codes  and  lists  individual  tasks  as  Mission  Performance 
Standards  (MPSs).  Using  the  events  of  the  MOC  as  a  model,  we  developed  a 
list  of  individual  tasks  to  train.  The  T  &  R  Manual  summarizes  convoy  operations 

77 


into  a  single  platoon  collective  task  INF-MOBL-5150  “Conduct  convoy 
operations.”  We  broke  this  task  down  into  subtasks  that  participants  could  easily 
evaluate.  The  T  &  R  Manual  supports  each  of  these  tasks,  although  we 
separated  some  subtasks  for  the  sake  of  evaluation  convenience.  Table  1 
shows  our  final  list  of  subtasks  and  traces  their  origins  in  the  T  &  R  Manual. 


Evaluated  Event  in  Experiment 

T  &  R  Manual  Source 

React  to  unexploded  Improvised 
Explosive  Device 

INF-MOBL-3150  React  to  an 
unexploded  Improvised  Explosive 

Device  (lED) 

React  to  an  Improvised  Explosive 

Device  detonation 

INF-MOBL-3151  React  to  an 

Improvised  Explosive  Device  (lED) 
detonation 

Take  immediate  action  against  a 
blocked  ambush 

INF-MOBL-5155  Take  immediate 
action  against  blocked  ambush  during 
motor  march 

Take  immediate  action  against  an 
unblocked  ambush 

INF-MOBL-5156  Take  immediate 
action  against  unblocked  ambush 
during  a  convoy 

Cordon  and  360  degree  security 

Component  event  9  of  INF-MOBL-5150 
Conduct  convoy  operations 

Employ  vehicle  machine  guns  / 
weapons 

INF-WPNS-5308  Conduct  heavy 
machinegun  offensive  operations 

Mounted  fire  and  maneuver 

INF-WPNS-5308  Conduct  heavy 
machinegun  offensive  operations 

Shift  fires  /  cease  fires 

INF-WPNS-5308  Conduct  heavy 
machinegun  offensive  operations 

Vehicle  recovery  /  bump  plan 

Component  events  11  through  15  of 
INF-MOBL-5150  Conduct  convoy 
operations 

Casualty  evacuation 

INF-MED-5430  Process  casualties 

Communication  with  higher 
headquarters 

Component  event  7  of  INF-MOBL-5150 
Conduct  convoy  operations 

Communication  between  vehicles  in 
convoy 

Component  event  7  of  INF-MOBL-5150 
Conduct  convoy  operations 

Communication  between  personnel  in 
vehicle 

Component  event  7  of  INF-MOBL-5150 
Conduct  convoy  operations 

Table  1 .  Evaluated  tasks  in  convoy  training  experiments 
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We  did  not  use  the  MOC  itself  in  this  study,  but  one  must 
understand  the  idea  of  the  exercise  as  background  to  this  work.  This  exercise 
provided  the  inspiration  for  the  scenarios  we  developed.  Also,  the  MOC  inspired 
the  live  exercise  used  as  an  evaluation  mechanism  in  a  follow-on  experiment. 
Our  analysis  of  this  exercise  included  personal  observation  and  detailed  analysis 
of  the  grading  sheet  used  to  evaluate  a  platoon’s  performance.  The  MOC  is 
developed  on  solid  Marine  Corps  doctrine  and  tactics,  techniques,  and 
procedures.  We  based  the  entire  body  of  work  discussed  in  this  thesis  on  a  real 
operational  exercise  conducted  by  each  unit  in  the  Marine  Corps  at  some  point  in 
the  training  cycle. 

b.  Background  Literature  Supports  Survey  Techniques 

The  ultimate  test  of  the  effectiveness  of  a  serious  game  as  a 
training  tool  is  improved  performance  in  the  real  world.  However,  this  approach 
is  not  the  most  practical  initial  step  in  the  laboratory  environment.  For  this 
reason,  the  bulk  of  the  study  of  serious  game  effectiveness  has  involved  user 
self-assessment.  The  Aussie  studies  of  VBS  2™  discussed  in  Chapter  II  provide 
some  of  the  most  valuable  published  insight  into  the  utility  of  VBS  2™  as  a 
training  tool.  Table  2  summarizes  all  of  the  previously  referenced  Aussie  VESL 
studies.  Table  2  highlights  a  trend  in  measurement  technique  across  the 
spectrum  of  the  research  work:  that  is,  a  consistent  use  of  participant 
questionnaires  to  determine  how  well  the  simulation  worked. 
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Year 

Project 

Independent 

Variable 

Dependent 

Variable 

Measurement 

Technique 

Title: 

-person  tactical  shooters:  COTS  games  with  military  training  potential? 

2002 

Determine 
whether  VBS  2™ 
had  military 
training  potential 

None 

None 

13  question  survey 
of  12  SME’s 

Title: 

Constructing  the  virtual  section 

2004 

Examine  decision 
making 
techniques 
employed  in 
game  play 

Amount  of 
verbal 

communication 

Use  of  roles 

Awareness  of 
team 

disposition 

Performance 

Points  scored  in 
game;  participant 
questionnaire 

Title: 

Factors  in  team  performance  in  a  vin 

tual  squad  environment 

2004 

Determine  the 
ideal  squad  size 

Size  of  section 

Performance 

Loss  Exchange 

Ratio;  participant 
questionnaire 

Title: 

The  use  of  games 

0  investigate  tactical  decision-making 

2005 

Examine  decision 
making  models 
employed  in 
game  play 

Decision 

making 

approach 

Performance 
in  training 
scenario 

Correlation: 
decision  making  by 
questionnaire, 
performance  by 
number  of  win/lose 
matches  won 

Title: 

Challenging  the  super  soldier  syndrome  in  1^^  person  simulations 

2005 

Examine  the 
extent  of  Super 
Soldier 

Syndrome  in 

VBS  2™ 

Use  of 
suppression; 

Use  of 
accuracy  / 
dispersion 

Change  in 
game  tactics 
(tempo  of 
movement, 
use  of  cover, 
firing 

procedure, 
personal 
tactics,  team 
tactics) 

Points  scored  in 
game;  participant 
questionnaire 
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Title:  Proficient  soldier  to  skilled  Gamer:  Training  for  COTS  success 

2005 

Determine  how 
long  it  takes  a 
user  to  be  able  to 
use  VBS  2™  as 
an  extension  of 
warfighting  skills 

Time  to  learn 
interface 

Performance 

Pass  a  “Test  of 
Objectives” 

Questionnaire 
asking  when 
participants  felt 
comfortable 

Title:  Heart-rate  and  immersion  in  a  first-person  simulation 

2006 

Determine 
physiological 
response  and 
sense  of 
presence  in  VBS 

2Tm 

Activity  (rest, 

walking, 

playing 

different 

scenarios) 

Physiological 
response  and 
sense  of 
presence 

Heart  rate  with  a 
monitor  and  7 
question  survey 

Title:  After  action  review  of  simulation  results:  impact  of  presentation  modality 

2006 

Examine  different 
visualization 
modalities  to 
enable  a 
commander  to 
effectively  use 

VBS  2™  for  AAR 

AAR 

presentation 

modality 

Utility  of 
modality 

Questionnaire 

Table  2.  Summary  of  performance  measurement  techniques 

used  in  Aussie  studies 


In  Chapter  II,  we  discussed  two  Naval  Postgraduate  School 
projects  involving  serious  games.  In  2005,  U.S.  Army  Majors  J.  Nolan  and  J. 
Jones  used  Delta  Force:  Black  Hawk  Down-Team  Sabre™  as  a  small  unit  tactics 
training  platform  at  the  Infantry  Officer  Basic  Course  in  Fort  Benning,  GA.  The 
two  researchers  compared  survey  data  from  a  control  group  who  did  not  use 
simulation  and  a  test  group  who  used  the  COTS  game  to  demonstrate  that 
serious  games  could  be  used  for  training  (Nolan  &  Jones,  2005).  In  2007,  U.S. 
Marine  Corps  Major  Neil  Fitzpatrick  and  Turkish  Army  Captain  Umit  Ayvaz  used 
a  similar  survey  mechanism  to  determine  the  effectiveness  of  a  decision-making 
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trainer  called  Close  Combat  Marine  (Fitzpatrick  III  &  Ayvaz,  2007).  In  both 
research  efforts,  user  feedback  and  self-assessment  provided  the  sole  basis  for 
effectiveness  analysis. 

In  summary,  relevant  literature  indicates  that  the  first  sensible  step 
in  determining  the  effectiveness  of  a  simulation  trainer  is  to  test  user  appreciation 
of  the  serious  game.  Previous  work  demonstrates  how  a  researcher  can 
leverage  survey  techniques  to  gain  insight  into  the  effectiveness  of  the  training 
tool.  We  proposed  to  conduct  a  small  unit  tactical  convoy  exercise  with  VBS 
and  see  how  participants’  perceptions  of  the  serious  game  and  their  own  tactical 
skill  changed. 


c.  Research  Interests 

As  a  pilot  study  supporting  a  larger  project,  this  project  served  two 
main  purposes.  First,  as  the  background  literature  indicates,  progress  in  future 
VBS  2™  effectiveness  evaluation  projects  requires  investigation  into  user 
perceptions  of  the  serious  game.  The  larger  supported  project  focused  on  small 
unit  convoy  tactics.  Thus,  this  project  started  with  the  development  of  tactical 
scenarios  and  tested  them  with  two  user  groups.  We  conducted  the  scenario 
testing  to  gain  insight  into  the  following  two  null  hypotheses: 

•  The  use  of  VBS  2™  does  not  change  the  appreciation  of 
serious  games  as  a  training  tool,  as  measured  by  the  Likert 
scaled  responses  of  participants. 

•  Participants  feel  that  the  use  of  VBS  2™  does  not  change 
their  small  unit  convoy  knowledge  and  skills,  as  measured 
by  Likert  scaled  responses. 

Second,  for  planned  future  projects,  groups  and  teams  will  use  VBS 
2™  as  a  common  environment  to  interact  toward  some  unit  goal.  As  in  the  case 
of  the  interface  pilot  study  discussed  in  the  previous  section,  a  lone  researcher  in 
a  laboratory  can  only  troubleshoot  a  limited  number  of  issues  in  support  of  such  a 
project.  Common  sense  dictates  that  many  unpredicted  problems  and  quirks  will 
arise  from  group  use  that  the  lone  laboratory  developer  can  never  find  by  himself. 
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In  this  regard,  this  pilot  study  served  as  the  first  effort  in  the  larger  project  of 
having  multiple  people  work  together  in  a  common  environment. 

The  TTECG  MOC  exercise  involves  Marines  patrolling  a 
designated  course  in  six  to  eight  vehicles  armed  with  heavy  weapons.  The 
course  trains  the  following  core  skills:  react  to  an  unexploded  improvised 
explosive  device  (lED),  react  to  an  lED  detonation,  take  immediate  action  against 
a  blocked  ambush  during  a  convoy,  take  immediate  action  against  an  unblocked 
ambush  during  a  convoy,  evacuate  a  damaged  vehicle,  and  evacuate  a  casualty. 
We  established  these  skill  sets  as  the  core  objectives  of  our  simulation  training. 
From  these  skills,  we  derived  a  series  of  three  progressively  more  difficult 
scenarios.  The  pilot  study  trained  participants  with  these  scenarios,  comparing 
their  attitude  toward  serious  games  and  their  self-assessed  tactical  skill  before 
and  after  the  training. 

2.  Method 

a.  Participants 

We  conducted  the  pilot  study  in  two  distinct  parts.  The  first  part. 
Experiment  2,  consisted  of  testing  a  single  convoy  scenario  in  a  school 
environment  at  the  Naval  Postgraduate  School  in  Monterey,  CA.  The  second 
part.  Experiment  3,  tested  all  three  scenarios  with  operational  Marines  from  a 
training  command  in  Camp  Pendleton,  CA.  Experiment  2  served  to  work  out  the 
technical  issues  in  running  the  scenario,  such  as  testing  radio  operation, 
ensuring  participants  matched  with  the  right  avatars  in  the  game,  and  checking 
networking  ideas  for  feasibility.  Experiment  3  gave  Marines  with  convoy 
experience  the  opportunity  to  comment  on  the  tactical  validity  of  the  scenarios 
and  VBS  2™  itself.  We  evaluated  both  groups  for  change  in  attitude  toward  the 
simulation  and  change  in  self-assessed  tactical  proficiency. 

Experiment  2  was  broken  into  four  50-minute  sessions  conducted 
weekly  for  one  month.  Twenty  Naval  Postgraduate  School  Modeling,  Virtual 
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Environment,  and  Simulation  (MOVES)  graduate  students,  ranging  in  age  from 
27  to  41  years,  participated  in  the  pilot  study.  All  participants  volunteered  as  part 
of  a  seminar  program  within  the  curriculum.  All  the  participants  were  male 
military  officers  with  service  times  ranging  from  4  to  24  years.  Three  foreign 
officers  participated,  and  U.S.  officers  represented  all  four  of  the  armed  services. 
The  first  session  provided  training  on  the  use  of  the  VBS  2™  interface  focusing 
on  the  following  tasks  deemed  applicable  to  the  convoy  scenarios:  individual 
body  movements,  manipulation  of  personal  gear,  use  of  personal  weapons, 
vehicle  interaction  and  operation,  use  of  vehicle  weapons,  manipulating  a 
casualty,  towing  a  vehicle,  disarming  an  lED,  and  recognition  of  friendly,  civilian, 
and  enemy  avatars.  The  second  session  provided  a  short  20-minute  overview  of 
convoy  tactics.  Participants  then  had  a  15-minute  practice  period  to  practice 
mounting  vehicles  in  the  assembly  area,  forming  into  march  formation,  tactical 
movement  in  a  benign  environment,  and  control  of  fires  on  a  target  array.  During 
the  third  and  fourth  sessions,  the  participants  conducted  a  tactical  convoy 
scenario. 

We  conducted  Experiment  3  in  a  full  day  of  training  on  site  at 
Weapons  and  Field  Training  Battalion  (WFTBN)  San  Diego  in  Camp  Pendleton, 
California.  Our  training  schedule  for  the  day  is  included  in  Appendix  K.  Twenty- 
four  Marines,  ranging  in  age  from  19  to  29  years,  participated  in  the  study.  Unit 
leadership  randomly  selected  eight  Marines  from  each  of  the  three  companies  for 
participation  in  the  training.  The  battalion  conducts  marksmanship  and  field 
training  for  recruits,  so  approximately  two-thirds  of  the  participants  were  infantry, 
and  the  other  third  were  support  personnel  including  communicators,  armorers, 
and  supply  and  administration  personnel.  The  training  group  consisted  of  one 
Private  (El),  one  Private  First  Class  (E2),  three  Lance  Corporals  (E3),  nine 
Corporals  (E4),  nine  Sergeants  (E5),  and  one  Staff  Sergeant  (E6).  The  rank 
structure  provided  us  the  ability  to  structure  the  personnel  like  a  typical  Marine 
platoon,  although  the  Marines  had  not  operated  together  before.  We  chose 
WFTBN  because  of  the  combat  experience  of  the  Marines  in  the  unit.  WFTBN  is 
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a  garrison  training  environment  that  serves  as  an  opportunity  for  Marines  to  get  a 
break  from  heavy  operational  deployment  cycles  in  the  fleet  forces.  Only  five  of 
the  Marines  had  never  been  to  combat.  Eighteen  Marines  had  participated  in 
convoy  operations  in  combat,  and  twelve  Marines  reported  more  than  100 
personal  convoy  operations  in  combat.  As  a  group,  the  WFTBN  Marines  were 
Subject  Matter  Experts  in  the  tactical  skills  included  in  this  study.  The  second 
study  included  the  same  exercise  progression  as  the  first.  Marines  started  with 
the  basic  interface  training  and  then  moved  on  to  the  progressively  more  difficult 
tactical  convoy  scenarios. 

b.  Apparatus 

The  project  used  twenty  Dell  Precision  M6300  laptops  from  a  suite 
of  the  Marine  Corps  Deployable  Virtual  Training  Environment  (DVTE)  package. 
Computers  were  networked  using  D-LINK  DGS-2208  8  port  switches.  Peripheral 
equipment,  including  mice,  cables,  and  switches  came  from  the  standard  DVTE 
package. 

We  used  a  standard  classroom  for  Experiment  2  with  the  MOVES 
students.  For  the  familiarization  training  and  evaluation,  we  networked  the 
computers  in  pairs  with  four  computers  per  switch.  Paired  users  faced  each 
other  at  a  classroom  table  so  that  they  could  not  see  each  other’s  screens.  For 
the  convoy  exercises,  we  connected  the  computers  on  a  single  network  with  six 
computers  per  switch.  Participants  sat  in  classroom  seating  in  a  theater-style 
classroom.  The  three  members  of  each  truck  crew  sat  in  a  row,  but  were 
dispersed  so  that  they  could  not  see  each  other’s  computer  screens.  The 
diagram  of  the  room  setup  is  included  in  Appendix  J.  Participants  communicated 
using  the  Joint  Virtual  Tactical  Radio  (JVTR),  a  voiceover  IP  system.  JVTR 
allows  the  use  of  two  communication  nets,  so  vehicle  commanders 
communicated  on  one  net,  while  the  other  net  was  used  as  a  vehicle  intercom  for 
the  truck  crew  to  communicate.  Figure  7  shows  the  setup  of  the  room  for  the 
convoy  exercises. 
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Figure  7.  Tactical  convoy  training  at  the  Naval  Postgraduate  School 


For  Experiment  3  conducted  at  WFTBN,  we  used  a  long,  narrow 
classroom  in  the  unit’s  academic  building.  The  facility  featured  overhead 
projection  support  and  had  more  than  enough  room  for  the  twenty-four 
participants  to  spread  out.  We  used  about  half  of  the  room’s  individual  desks 
and  interspersed  them  around  the  room  to  seat  participants  in  three-person 
groups  with  maximum  space  between  groups.  We  used  a  setup  similar  to  the 
MOVES  study  with  computers  networked  at  five  to  eight  per  switch,  depending 
on  the  physical  locations  of  the  machines.  Generally,  participants  could  not 
easily  see  each  other’s  screens,  although  the  setup  did  not  prohibit  the 
determined  Marine  from  doing  so.  We  used  the  same  system  for  JVTR 
communications  that  we  used  in  the  MOVES  study,  with  two  radio  nets 
supporting  vehicle  and  convoy  communications,  respectively.  Figure  8  shows 
the  WFTBN  training  site. 
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Figure  8. 


Tactical  convoy  training  at  WFTBN,  San  Diego 


c.  Procedure 

We  used  a  similar  experiment  procedure  for  both  the  MOVES  and 
WFTBN  experiments.  Before  the  tactical  scenario  training,  participants 
completed  a  two-page  questionnaire,  included  in  Appendix  L.  The  questionnaire 
included  demographic  and  computer  usage  information.  This  questionnaire  also 
included  a  set  of  questions  intended  to  determine  the  participant’s  attitude  toward 
personal  computer-based  simulation  training.  Another  set  of  questions 
determined  the  participants’  perception  of  tactical  skill.  Responses  to  both  sets 
of  questions  were  recorded  on  a  5-point  Likert  scale.  After  the  tactical  scenario 
training,  participants  completed  a  three-page  post-exercise  questionnaire, 
included  in  Appendix  N,  that  featured  subjective  questions  about  the 
effectiveness  of  the  training.  The  simulation  attitude  questions  and  tactical 


proficiency  questions  from  the  pre-exercise  questionnaire  were  repeated  in  the 
second  questionnaire.  Comparison  of  these  Likert  scaled  responses  served  as 
the  basis  of  analysis  for  this  project. 

We  wanted  to  evaluate  an  academic  test  for  later  use,  and  we 
administered  the  test  to  the  WFTBN  group.  We  developed  two  similar  tests  with 
eighteen  multiple  choice  questions  each.  The  questions  related  to  the  general 
tactical  areas  of  reaction  to  ambush,  reaction  to  lED,  casualty  evacuation,  and 
vehicle  recovery.  Each  of  the  questions  had  four  choices  for  an  answer  with  one 
best  answer.  We  developed  the  tests  from  the  training  materials  administered  for 
convoy  classes  at  TTECG  in  Twentynine  Palms  and  The  Basic  School  in 
Quanico,  VA,  that  trains  all  Marine  Corps  junior  officers.  We  designed  the  tests 
to  be  similar  but  different  with  paired  questions.  For  example,  a  reaction  to 
ambush  question  on  one  test  would  have  a  similar  question  on  the  other  test. 
The  Marines  took  one  of  the  tests  before  training  and  one  after  training.  We 
randomized  the  test  taken  first,  with  half  taking  one  of  the  tests  first  and  the  other 
half  taking  the  other  test  first.  We  included  both  tests  in  Appendices  P  and  Q. 

For  all  tactical  scenarios,  participants  were  grouped  in  teams  of 
three:  a  vehicle  commander  riding  in  the  passenger  seat,  a  driver,  and  a  gunner 
for  the  vehicle’s  weapon  system.  Seven  vehicles  participated  in  the  MOVES 
exercise,  and  eight  vehicles  participated  in  the  WFTBN  exercise.  The  vehicles 
consisted  of  a  mix  of  five  HMMWVs  and  two  MTVRs  armed  with  an  even  mix  of 
0.50  caliber  machine  guns  and  MK  19  40  mm  grenade  launchers.  Each  tactical 
scenario  consisted  of  a  series  of  stations,  including  reaction  to  an  inert  I  ED, 
vehicle  recovery,  reaction  to  an  unblocked  ambush,  casualty  evacuation,  and 
reaction  to  a  blocked  ambush.  The  scenarios  used  the  game’s  Twentynine 
Palms  database  and  involved  operation  in  the  western  Quackenbush  area  of  the 
terrain.  The  scenarios  varied  from  roughly  20  kilometers  in  length  to  a  scenario 
involving  more  than  40  kilometers  of  the  western  side  of  the  Twentynine  Palms 
training  area. 
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For  analysis,  we  used  statistical  group  comparison  techniques 
including  the  paired  t-test  and  the  Wilcoxon  rank-sum  method.  For  all  statistical 
tests,  we  used  an  alpha  level  of  0.05. 

3.  Results 

First  and  foremost,  this  pilot  study  served  to  identify  unforeseen  problems 
in  the  project  methodology.  This  pilot  study  accomplished  this  goal  with  the 
following  observations; 

•  Some  users  failed  to  take  training  seriously  (negligent  discharges  / 
fratricide).  This  degraded  the  experience  for  the  whole  group,  not 
just  the  individual.  This  problem  was  much  more  pronounced  with 
the  WFTBN  group. 

•  Communication  was  a  burden  as  VBS  2™  has  no  internal 
communication  tool. 

•  Computer  glitches  hindered  the  commencement  of  training. 

•  12  computers  =  1  man-hour  setup  +  1  man-hour  startup  +  1  man¬ 
hour  breakdown;  It  takes  a  working  knowledge  to  set  up  the  system 
and  run  the  program  correctly 

While  these  unexpected  problems  confounded  the  data  collected  from  the 
evaluation  exercise,  the  survey  results  provided  valuable  feedback. 

a.  Survey  Results 

The  post-exercise  survey  implied  that  VBS  2™  had  little  training 
value  as  far  as  both  MOVES  students  and  WFTBN  Marines  were  concerned. 
MOVES  students,  having  little  to  no  convoy  experience,  suggested  that  learning 
immediate  action  drills  was  the  only  value  provided  by  the  game.  The  Marines, 
when  tested,  suggested  the  same  thing.  While  the  sample  was  relatively  small 
and  the  Marines  used  for  the  exercise  were  part  of  a  training  battalion,  over  half 
of  the  Marines  used  were  from  the  infantry  occupational  specialty.  Also,  most  of 
the  Marines  had  average  to  extensive  convoy  experience  and  could  be 
considered  “Subject  Matter  Experts”  in  convoy  standard  operating  procedure  and 
immediate  action  drills.  When  the  entire  sample  of  post-exercise  results  was 
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evaluated,  there  appeared  to  be  no  significance  that  would  lead  to  rejecting 
either  hypothesis,  implying  that  there  was  no  training  value  in  using  VBS  2™. 
However,  when  the  data  of  Marines  with  little  to  no  convoy  experience  was 
separated  from  the  whole  to  more  closely  mirror  the  sample  from  the  NPS  trials, 
we  found  significance  in  some  of  the  immediate  action  drills  just  as  we  did  from 
the  MOVES  students. 

b.  Academic  Knowledge  Test  Results 

Only  the  WFTBN  study  involved  the  administration  of  the  academic 
knowledge  test.  In  this  endeavor,  we  wanted  to  ascertain  whether  the  test  made 
tactical  sense  to  our  pool  of  Subject  Matter  Experts.  We  did  not  give 
consideration  to  the  results  of  the  academic  knowledge  test  as  an  indicator  of  the 
potential  effectiveness  of  the  simulation  training  in  this  study  because  we  did  not 
design  the  study  to  improve  academic  knowledge  of  convoy  operations.  Rather, 
the  study  served  to  determine  whether  the  scenarios  we  had  developed  met  our 
technical  and  tactical  expectations.  With  this  in  mind,  our  analysis  of  the 
academic  knowledge  test  results  did  not  focus  on  change  in  performance. 
Instead,  we  concerned  ourselves  with  the  fairness  of  the  test,  and  whether  the 
administration  of  the  test  produced  data  that  could  be  analyzed  in  later  studies. 

We  recorded  the  results  of  the  test  in  a  spreadsheet.  If  Marines 
answered  questions  incorrectly,  we  recorded  the  incorrect  response  in  the 
corresponding  spreadsheet  cell.  With  this  data,  we  not  only  examined  how  many 
Marines  missed  each  question,  but  whether  the  questions  were  answered 
incorrectly  with  consistent  responses.  In  this  way,  we  could  generally  group 
questions  into  three  categories:  questions  that  were  answered  correctly, 
questions  in  which  Marines  could  not  agree  on  an  answer  (incorrect  and 
inconsistent  answers),  and  questions  in  which  Marines  as  a  group  disagreed  with 
our  answer  (incorrect,  but  consistent  answers).  The  last  two  categories  of 
questions  differed.  Incorrect  and  inconsistent  answers  indicated  that  Marines  did 
not  understand  the  material  addressed  by  the  question.  Incorrect  and  consistent 
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answers  indicated  that  the  Marines  as  a  group  had  an  understanding  of  the 
question  but  looked  at  the  issue  differently  than  we  intended.  Most  likely,  this 
final  problem  suggested  a  problem  with  the  wording  of  the  question. 


Tables  3  and  4  summarize  the  data  from  the  knowledge  tests.  We 
started  by  highlighting  questions  that  we  considered  a  problem,  and  we  used  a 
criterion  of  less  than  60%  of  the  Marines  answering  the  question  correctly.  Using 
this  idea,  we  noted  that  Test  1  had  six  problem  questions  while  Test  2  had  eight. 
Next,  we  analyzed  the  responses  to  look  for  problem  questions  that  may  have 
resulted  from  unclear  wording.  We  noted  that  Question  4  from  Test  1  and 
Questions  3,  13,  and  16  from  Test  2  particularly  fit  in  this  category.  These  four 
questions  were  noteworthy  because  they  all  related  to  organization  of  the  convoy 
and  division  of  labor  within  the  convoy  organization. 


Question 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

Number  respondents  for  each  answer  Yellow  highlight  indicates  the  correct  response 

A 

0 

3 

8 

14 

21 

15 

2 

6 

3 

21 

7 

2 

4 

0 

22 

20 

5 

7 
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20 

10 

1 

4 

2 

3 

4 

1 

10 

0 

15 

5 

8 

1 

1 

1 

9 

4 

c 

0 

3 

4 

3 

1 

0 

1 

2 

5 

0 

1 

13 

5 

18 

0 

1 

5 

0 

D 

4 

7 

10 

2 

0 

6 

17 

15 

6 

2 

0 

2 

5 

4 

0 

1 

4 

11 

Other 

0 

1 

1 

1 

0 

0 

0 

0 

0 

1 

1 

2 

2 

1 

1 

1 

1 

2 

Percentage  for  each  question 


Yellow:  no  concern,  red:  concern  (criteria  60%) 


0.21 

0.25 


A 

0.00 

0.13  1  0.35 

0.61 

0.88 

0.63 

0.08 

0.25 

B 

0.83 

0.04 

0.17 

0.08 

0.13 

0.17 

0.04 

c 

0.00 

0.13  1  0.17 

■ 

0.04 

0.00 

0.04 

0.08 

D 

0.17 

0.30 

0.09 

0.00 

0.25 

0.71 

0.63 

0.91 

0.00 

0.00 

0.09 


0.09 

0.23 


0.30 
0.65 
0.04 
0.00 


0.18 

0.36 


0.00 

0.04 

0.78 

0.17 


0.96 

0.04 

0.00 

0.00 


0.87 

0.04 

0.04 

0.04 


0.22 

0.39 


0.32 

0.18 


0.17 


Table  3.  Analysis  of  respondent  answers  to  Academic  Knowledge  Test  1 
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Question 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

Number  respondents  for  each  answer  Yellow  highlight  indicates  the  correct  response 

A 

5 

6 

15 

10 

4 

0 

1 

1 

18 

18 

2 

4 

14 

4 

5 

22 

13 

0 

B 

3 

16 

3 

6 

12 

21 

3 

19 

2 

0 

1 

6 

1 

5 

1 

1 

3 

9 

c 

14 

1 

2 

6 

3 

0 

2 

0 

2 

2 

16 

12 

1 

13 

2 

0 

2 

2 

D 

1 

0 

3 

1 

4 

1 

17 

3 

1 

2 

4 

0 

5 

1 

15 

0 

5 

12 

Other 

1 

1 

1 

1 

1 

2 

1 

1 

1 

2 

1 

2 

3 

1 

1 

1 

1 

1 

Percentage  for  each  question 


Yellow:  no  concern,  red:  concern  (criteria  60%) 


A 

0.22 

B 

0.13 

c 

0.61 

D 

0.04 

Table  4.  Analysis  of  respondent  answers  to  Academic  Knowledge  Test  2 


c.  Analysis 

The  design  of  the  paired  before  and  after  surveys  dictated  an 
analysis  of  paired  Likert  scale  responses.  That  is,  we  looked  at  the  differences 
between  numerical  responses  to  the  attitude  questions  and  skill  sets  for  changes 
in  attitude  or  proficiency.  We  first  analyzed  the  survey  data  to  find  any  linear 
dependence  (multicolinearity)  between  the  survey  questions  that  would  indicate 
that  the  answer  to  a  specific  question  would  influence  the  answer  to  one  or  more 
others.  To  evaluate  this  we  used  a  pair  plot  of  the  responses  to  all  questions  on 
the  “after”  survey,  as  shown  in  Figure  9.  Not  surprisingly,  this  pair  plot  showed  a 
strong  linear  relationship  between  all  four  questions  which  indicated  that  they 
together  shared  some  attribute(s)  of  the  serious  gaming  that  the  participants  felt 
would  be  valuable  in  unit  training.  We  found  similar  linear  relationships  to  be 
strong  in  other  questions  pertaining  to  self-assessment. 
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Figure  9.  Pair  plot  of  survey  questions  related  to  user  appreciation  of 

simulation  training 

Next,  we  used  a  Wilcoxon  Rank  test  to  determine  which  responses 
had  changed  with  statistical  significance.  For  the  MOVES  students  who  trained 
in  the  first  part  of  the  study,  we  found  three  tactical  skills  to  have  significance  as 
shown  in  Table  5.  The  MOVES  students,  representing  untrained  personnel  in  a 
non-Marine  setting  with  respect  to  convoy  experience,  responded  that  VBS  2™ 
improved  knowledge  and  ability  of  immediate  action  to  an  unblocked  ambush, 
cordon  and  360°  security,  and  casualty  evacuation. 
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4 

Unblocked 

Ambush  lA 

-2.9266 

0.0017 

5 

Cordon  and  360 
degree  security 

-2.0991 

0.0179 

10 

Casualty 

evacuation 

-2.1297 

0.0166 

Table  5.  Self-assessment  questions  that  showed  a  significant  difference  in 

response  for  MOVES  students 

When  we  conducted  the  similar  exercise  at  WFTBN  and  a  group  of 
Marines  with  mixed  convoy  experience  was  tested,  we  found  three  tactical  skills 
to  have  significant  change,  as  shown  in  Table  6.  None  of  the  skills  was  common 
to  those  that  showed  significant  change  in  the  MOVES  group.  It  is  important  to 
note  that  these  results  were  only  significant  for  Marines  who  had  little  or  no 
convoy  experience.  The  Marines  who  had  more  than  twenty-five  convoys  found 
that  the  training  provided  little  or  no  value  and  actually  degraded  their  self- 
assessed  performance  in  some  cases. 


Table  6. 


1 

React  to 

unexploded  lED 

2.1503 

0.0158 

2 

React  to 

detonated  lED 

2.0827 

0.0186 

6 

Employ  weapons 

2.1333 

0.0164 

Self-assessment  questions  that  showed  a  significant  difference  in 
response  for  WFTBN  Marines 
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4. 


Discussion 


a.  Initial  Findings 

The  series  of  initial  pair  plots  revealed  strong  linear  relationships 
among  aspects  of  the  game  experience,  as  well  as  the  participants’  assessment 
of  their  level  of  skill  and  familiarity  with  convoy  tactics.  These  strong  linear 
relationships  led  us  to  investigate  what  aspects  of  the  game,  or  tactical  skills, 
affected  the  Likert  scaled  responses  most.  Having  had  each  participant  answer 
detailed  pre-  and  post-training  surveys,  we  then  made  a  data  set  that  enabled  us 
to  review  how  the  delta  of  each  question  was  affected  by  the  training. 

The  Wilcoxon  Rank  tests  showed  that  VBS  2™  did  not  significantly 
change  participants’  attitudes  or  appreciation  of  serious  gaming.  This  is  an 
interesting,  if  not  surprising,  result,  given  that  the  relative  youth  of  our  participants 
did  not  translate  into  excitement  or  increased  appreciation  for  serious  gaming. 
The  resulting  p-value  of  this  assessment  meant  that  there  was  no  basis  to  reject 
our  first  null  hypothesis:  The  use  of  VBS  2™  does  not  change  the  appreciation 
of  serious  games  as  a  training  tool  as  measured  by  the  Likert  scaled  responses 
of  participants. 

b.  Digging  Deeper 

The  failure  to  reject  our  first  null  hypothesis  did  not  necessarily 
mean  that  VBS  2™  was  devoid  of  any  benefits  toward  training.  We  examined 
self-assessed  proficiency  to  determine  if  participants  felt  that  their  tactical  abilities 
had  changed  even  if  their  opinion  of  simulation  as  a  training  device  had  not. 
Knowing  that  our  participants  would  have  various  military  skills,  experiences  and 
time  in  the  military,  we  structured  the  pre-  and  post-training  surveys  to 
incorporate  these  factors.  By  cross-examining  military  experience  with  questions 
that  elicited  our  participants  to  assess  their  personal  proficiency  in  immediate 
action  drills  (such  as  their  confidence/ability  to  take  action  against  a  blocked 
ambush),  we  obtained  two  revealing  solution  sets  to  Wilcoxon  Rank  tests: 
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•  MOVES  students  who  reported  themselves  as  having  little 
convoy  knowledge  and  experience  reported  that  VBS  2™ 
improved  their  knowledge,  ability  and  confidence  to  take 
immediate  action  in  the  cases  of  an  unblocked  ambush, 
cordon,  and  establish  360°  security  and  conduct  casualty 
evacuation. 

•  Among  the  operational  Marines  who  participated  in  the 
testing  at  WFTBN,  those  who  reported  having  no  practical 
convoy  experience  felt  that  VBS  2™  improved  their 
knowledge,  ability  and  confidence  to  take  immediate  action 
in  the  cases  of  an  unexploded  lED,  detonated  lED,  and  in 
the  tactical  employment  of  weapons. 

These  tests  suggest  that  VBS  2^''^  might  be  an  effective  introductory 
training  tool  for  those  who  have  little  or  no  convoy  experience.  Where  actual 
convoy  rehearsals  and  training  may  be  difficult  or  burdensome,  VBS  2™  seems 
to  be  a  clear  alternative  to  expose  junior  Marines  to  training  at  the  schoolhouse 
or  first  duty  station. 

We  examined  the  six  skills  for  which  participants  identified  an 
increase  in  self-assessed  proficiency.  We  could  find  no  reason  that  would 
explain  why  the  MOVES  group  showed  a  change  in  three  skills  and  the  WFTBN 
novice  users  showed  a  change  in  three  separate  skills.  The  groups  certainly 
differed,  but  the  differences  did  not  relate  to  the  skills  in  any  way.  From 
observation  of  the  training  in  progress,  we  offer  that  participants  might  tend  to 
indicate  a  change  in  self-assessed  proficiency  for  the  exercises  that  go  well. 
Some  of  the  exercises  experienced  difficulties,  such  as  communications 
problems,  computer  glitches,  and  internal  command  and  control  problems.  On 
the  other  hand,  other  parts  of  the  exercise  went  smoothly  so  that  standard 
procedures  were  executed  as  intended.  We  suggest  that  the  particular  skill 
assessed  as  improved  might  relate  more  to  what  goes  well  than  to  any  other 
factor.  If  novice  trainees  see  a  smooth  execution,  then  they  may  feel  more 
proficient  in  that  particular  skill. 


96 


c. 


Applicable  for  Even  the  Veteran  Marine 


For  the  veteran  Marines  who  reported  that  VBS  2™  may  have 
actually  degraded  their  skill  or  confidence  in  their  convoy  ability,  we  recommend 
further  study  and/or  examination  of  the  following  factors: 

•  VBS  2™  was  not  employed  in  the  context  that  a  veteran 
Marine  might  see  with  his  actual  unit.  Such  a  future 
occasion  might  entail  them  virtually  driving  the  same  convoy 
route  prior  to  the  actual  convoy,  where  speeds,  terrain 
features  and  distances  would  all  serve  as  useful,  albeit 
virtual,  exposure  to  the  actual  route.  This,  in  essence,  would 
afford  them  to  “conduct  the  convoy  prior  to  conducting  the 
convoy.” 

•  Those  who  reported  a  degradation  in  skill  may  have  actually 
only  been  made  aware  of  their  “rustiness”  or  inflated  sense 
of  personal  mastery  of  what  it  is  to  conduct  a  convoy.  In  this 
sense,  perhaps  VBS  2™  serves  as  a  very  useful  reality 
check. 

To  this  end,  the  results  of  the  extended  study  of  the  responses  to  the 
pre-  and  post-training  surveys  led  to  our  conclusion  that,  in  certain  cases,  we  could 
reject  our  second  null  hypothesis.  Participants  feel  that  the  use  of  VBS  2™  does 
not  change  their  small-unit  convoy  knowledge  and  skills  as  measured  by  Likert 
scaled  responses.  The  instances  where  the  Wilcoxon  Rank  tests  yielded  p-values 
supporting  such  rejection  of  the  null  hypothesis  can  be  summarized  as  those 
involving  either  novice  convoy  trainees  or  exposure  to  immediate  action  drills. 

d.  Practical  Tips  for  Future  Studies 

VBS  2™  is  one  of  many  in  a  wide  spectrum  of  serious  games  that 
have  either  already  been  fielded  or  are  currently  in  development.  The  use  of 
VBS  2™  and  other  similar  serious  games  as  a  device  to  expose  junior 
servicemen  to  tactical  scenarios  is  clearly  fertile  ground  for  further  study. 

In  addition  to  determining  how  VBS  2™  affected  participant 
knowledge  and  confidence,  these  two  studies  also  provided  an  opportunity  to 
investigate  the  practical  aspects  of  conducting  training  using  the  DVTE  suite. 
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The  Marine  Corps  intends  DVTE,  including  VBS  2™,  to  be  a  portable  resource 
for  every  infantry  battalion.  Because  VBS  2™  is  intended  to  be  a  portable 
resource,  we  found  that  it  will  be  imperative  to  every  unit  to  appreciate  the 
following: 

•  Participants  must  take  the  training  seriously.  Negligent 
discharges  and  fratricide  severely  hinder  the  effectiveness 
and  spirit  of  the  training. 

•  The  length  of  training  and  the  days  on  which  it  is  scheduled 
(WFTBN  training  occurred  on  Friday)  ought  to  be  carefully 
considered  to  enable  participants  to  more  fully  focus  on  the 
training  at  hand. 

•  The  current  communications  package  (JVTR)  is  a  burden. 
Possible  replacement,  enhancements,  or  unit  workarounds 
need  to  be  considered.  The  optimal  solution  is  an  internal 
communication  package  within  VBS  2™  itself. 

•  A  larger  library  of  terrain  needs  to  be  developed  quickly. 
Areas  such  as  Twentynine  Palms  need  improved  fidelity. 
The  size  of  the  Twentynine  Palms  database  makes  it 
unwieldy.  For  small  unit  tactical  exercises,  fidelity  is  more 
important  than  terrain  size. 

•  The  computers’  operating  systems  have  recurring  glitches 
that  will  need  to  be  recorded  in  order  to  develop  patches  or 
user-level  workarounds  that  will  support  continued  training. 

e.  Conclusion 

The  objective  of  this  research  was  to  evaluate  user  perceptions  of 
the  serious  game  VBS  2™.  As  a  pilot  study,  it  was  also  the  first  effort  in  having 
multiple  people  work  together  in  a  common  virtual  environment.  We  met  both 
objectives  and,  moreover,  conclusive  data  led  us  to  the  fact  that  in  this  case,  a 
broad  hypothesis  could  not  be  applied  to  our  diverse  sample  population. 
Through  data  analysis  and  cross-examination  of  several  key  factors  (i.e.,  age, 
rank,  convoy  experience),  we  were  able  to  determine  that  the  perceived 
effectiveness  of  VBS  2™  as  a  training  tool  was  much  larger  than  a  single  Likert 
scaled  response.  In  fact,  as  a  training  tool,  it  holds  the  potential  to  be  applicable 
to  junior  and  veteran  servicemen  alike,  albeit  in  different  ways. 


98 


IV.  EXPERIMENT  IN  SUPPORT  OF  HYPOTHESIS  1 


A.  INTRODUCTION 

Experiments  1,  2,  and  3  prepared  us  to  use  VBS  2^''^  to  train  Marines. 
Experiment  1  demonstrated  sound  practices  to  familiarize  users  with  the  VBS  2™ 
interface  so  that  they  could  use  the  simulation  as  an  extension  of  their 
warfighting  skills.  Experiments  2  and  3  provided  a  forum  to  test  scenarios  to 
determine  what  worked  and  what  did  not.  Armed  with  this  information,  we  could 
address  the  questions  of  this  thesis  project.  With  this  in  mind,  we  designed 
Experiment  4  to  address  the  first  thesis  research  question: 

•  Does  preliminary  training  with  VBS  2™  positively  impact  performance 
in  a  platoon-level  tactical  scenario? 

In  our  consideration  of  the  task  of  training  convoy  operations  with  virtual 
environments,  we  noted  that  different  approaches  could  influence  the 
effectiveness  of  the  training.  A  study  at  the  Israeli  Air  Force  flight  school  in  1993 
inspired  our  choice  of  approach.  In  that  study,  cadets  used  a  computer  game 
called  Space  Fortress  II  in  the  early  stages  of  flight  training.  One  group  played 
seven  part-task  games,  one  at  a  time,  receiving  feedback  after  each.  The  other 
group  used  a  full  mission  game  with  all  tasks  covered  in  the  same  game.  A 
control  group  trained  with  traditional  techniques.  The  study  found  that  both 
simulation  groups  performed  better  in  subsequent  real-world  flights  than  the 
control  group,  and  the  part-task  group  outperformed  the  full  mission  group 
(Gopher  et  al.,  1994).  For  our  work,  we  considered  two  different  arrangements  of 
the  skill  sets  to  be  trained.  In  one  application,  a  Marine  trained  each  skill 
individually  in  depth,  presenting  progressively  more  difficult  training  for  each  skill 
individually  before  moving  to  the  next  skill.  The  second  approach  combined  all 
the  skills  into  a  single  training  evolution  and  developed  progressively  more 
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difficult  training  for  all  the  skills  as  a  group.  We  were  interested  in  whether  one  of 
these  approaches  would  produce  better  results,  so  we  developed  the  second 
research  question  for  Experiment  4: 

•  Is  there  a  difference  between  using  a  full  mission  approach  versus 
a  part-task  approach  to  training  when  using  VBS  2™  for  platoon- 
level  tactical  training? 

In  considering  the  design  of  this  experiment,  we  relied  heavily  on  previous 
student  work  at  the  Naval  Postgraduate  School.  The  previously  mentioned  work 
of  U.S.  Army  Majors  J.  Nolan  and  J.  Jones  with  Delta  Force:  Black  Hawk  Down- 
Team  Sabre™  served  as  a  basis  for  studying  how  gaming  technology  could 
support  training  goals  (Nolan  &  Jones,  2005).  Likewise,  Marine  Corps  Major  Neil 
Fitzpatrick  and  Turkish  Army  Captain  Limit  Ayvaz  worked  with  a  decision-making 
trainer  called  Close  Combat  Marine,  and  their  work  provided  insight  into  Best 
Practices  for  leveraging  surveys  and  questionnaires  to  gain  insight  into  user 
performance  (Fitzpatrick  III  &  Ayvaz,  2007).  We  summarized  the  body  of  work 
done  by  the  Aussie  Virtual  Environments  &  Simulation  Laboratory  (VESL)  with 
VBS  2™  in  Chapter  II  of  this  thesis.  All  of  these  research  efforts  directly 
contributed  to  the  design  of  Experiment  4. 

In  fact.  Experiment  4  is  similar  enough  to  some  of  these  previous  efforts 

that  it  may  seem  like  just  another  repetition  of  the  same  work.  For  Experiment  4, 

we  felt  it  was  crucial  to  conduct  the  study  with  an  operational  unit  preparing  for 

real-world  contingencies.  We  sought  to  take  the  personal  computer-based  game 

beyond  the  walls  of  the  schoolhouse  and  the  clear-cut  environs  of  the  university 

laboratory.  Experiment  3,  with  its  work  with  infantry  Marines  at  Weapons  and 

Field  Training  Battalion,  demonstrated  that  enlisted  Marines  with  a  “day  job”  in 

the  operational  forces  might  perceive  personal  computer-based  game  training 

differently.  For  this  reason,  we  expended  much  effort  to  establish  an  experiment 

in  the  operational  forces.  With  this  in  mind,  two  competing  factors  drove 

Experiment  4:  the  scientific  rigor  associated  with  any  hypothesis  test  and  the 

training  needs  of  an  operational  unit  very  much  involved  in  preparing  to  fight  a 

real-world  war.  Throughout  the  research  project,  we  placed  equal  emphasis  on 
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both  concerns.  That  is,  we  sought  soundly  and  scientifically  to  gain  insight  into 
the  thesis  question  at  hand,  but  we  also  sought  to  train  Marines  as  best  we  could 
with  the  tool  used  for  the  study.  These  Marines  needed  the  training  we  offered, 
and  failure  to  provide  it  was  unacceptable. 

We  chose  a  reserve  Marine  infantry  battalion  preparing  for  deployment  to 
combat  theater  for  Experiment  4.  In  part,  timing  drove  this  decision,  because  this 
particular  battalion  happened  to  be  in  the  preparation  phase  at  the  right  time  for 
this  research  project.  However,  other  factors  drove  this  decision  as  well. 
Reserve  Marines  typically  do  not  enjoy  the  benefits  of  simulation  available  to 
active  duty  Marines.  Reserve  units  dot  the  U.S.  map  with  no  regular  proximity  to 
active  duty  training  installations.  While  reserve  units  are  eligible  for  the 
Deployable  Virtual  Training  Environment  (DVTE)  described  earlier,  they  have  a 
lower  priority  for  the  equipment  than  their  active  duty  counterparts.  In  fact,  this 
particular  unit  had  been  working  to  get  the  gear  prior  to  their  deployment,  but  at 
the  time  they  left  home  site,  no  DVTE  equipment  had  been  used.  Because 
reserve  units  come  from  a  single  geographic  location,  they  typically  enjoy  better 
personnel  stability  than  active  duty  units.  Additionally,  reserve  Marines  often  stay 
in  the  same  platoons  and  companies  for  their  full  enlistment,  and  some  may 
spend  entire  careers  in  the  same  unit.  We  viewed  this  unit  cohesion  as  a 
positive  contributor  to  our  research  plans.  This  symbiotic  relationship  of  a  unit 
with  a  training  need  and  a  research  project  with  a  need  for  a  unit  promised 
mutual  benefit. 

In  the  end,  we  developed  an  experiment  design  that  was  a  compromise 
result  of  the  competing  needs  of  science  and  operational  training.  We  conducted 
our  research  with  one  of  the  battalion’s  infantry  companies.  We  conducted  the 
simulation  training  at  a  phase  in  the  battalion’s  pre-deployment  preparations  in 
which  the  unit  had  deployed  from  home  site  to  Camp  Pendleton,  California.  We 
had  a  week  of  training  “white  space,”  or  otherwise  unobligated  training  time,  to 
conduct  training  in  support  of  our  project.  During  this  week,  platoons  conducted 
training  in  support  of  the  study  for  four  hours  per  day  for  a  four-day  training  week. 
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The  particular  company  selected  for  the  research  had  a  live  exercise  aboard 
Camp  Pendleton  scheduled  that  would  serve  as  an  evaluation  metric  for  the 
study.  Within  these  parameters,  we  developed  Experiment  4  using  lessons 
learned  from  the  work  mentioned  above. 

B.  METHOD 

1.  Participants 

The  Weapons  Company  of  the  reserve  infantry  battalion  participated  in 
Experiment  4.  For  convoy  operations,  the  Weapons  Company  was  already 
divided  into  three  Motorized  Assault  Platoons  (MAPs),  and  these  MAPs  served 
as  the  three  treatment  groups  for  the  experiment.  A  Marine  Corps  Weapons 
Company  includes  infantry  with  a  variety  of  specialized  skills,  including  heavy 
machine  guns,  anti-tank  missiles,  medium  mortars,  and  other  heavy  weapons 
organic  to  an  infantry  battalion.  Many  of  these  weapons,  such  as  the  0.50  caliber 
machine  gun,  the  MK  19  automatic  grenade  launcher,  and  the  Tube-launched 
Optically-tracked  Wire-guided  (TOW)  missile  are  often  vehicle-mounted  for 
infantry  battalion  operations.  Therefore,  mounted  operations,  including  the 
convoy  mission  of  moving  men  and  materiel  from  one  place  to  another,  were 
standard  routine  training  objectives  for  the  Weapons  Company.  Moreover, 
proficiency  in  this  domain  would  certainly  be  a  demand  upon  deployment  to 
combat  theater. 

Most  of  the  Marines  in  each  MAP  were  infantry,  including  machine 
gunners  (MOS  0331),  mortarmen  (MOS  0341),  and  anti-tank  missilemen  (MOS 
0352).  However,  at  the  time  of  the  experiment,  the  ranks  of  the  infantry  battalion 
were  being  fleshed  out  with  a  detachment  from  a  communications  battalion,  so 
each  MAP  had  a  few  communicators,  including  field  radio  operators  (MOS  0621) 
and  ground  communications  repairmen  (MOS  2844)  sprinkled  into  the  platoon 
organization.  Each  MAP  varied  in  size,  but  twenty-four  Marines  from  each  MAP 
participated  in  the  study.  Unit  leadership  randomly  chose  these  Marines  with  no 
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input  or  cognizance  on  the  part  of  the  research  team.  The  unit  emphasized  intact 
vehicle  crews  participating  in  the  study  so,  for  the  most  part,  the  participants 
were  vehicle  crewmembers  identified  at  home  site  by  the  Company  Commander 
well  prior  to  the  execution  of  the  experiment.  With  this  in  mind,  we  assumed  that 
there  were  no  biases  in  the  selection  of  the  training  audience;  rather,  the 
Company  Commander  crafted  the  training  audience  from  the  stable  crews  early 
in  the  planning  process. 

Each  MAP  had  a  Lieutenant  serving  as  the  Platoon  Commander,  and  for 
Experiment  4,  these  Marines  served  as  the  trainers.  In  this  capacity,  they 
organized  the  participants,  supervised  exercise  briefing,  participated  in  using  the 
simulation  exercise  to  meet  specific  platoon  and  company  training  goals,  and 
conducted  the  debriefing  after  each  exercise.  The  Platoon  Sergeant  for  each 
MAP,  the  senior  enlisted  Marine  in  the  unit,  did  not  participate  in  the  training.  All 
three  MAPs  were  larger  than  the  24-man  training  audience  accommodated  in  the 
virtual  training,  so  the  Platoon  Sergeant  handled  the  Marines  who  were  not  being 
trained  in  the  experiment.  Each  MAP  had  an  enlisted  Marine  who  served  as  the 
Convoy  Commander  for  the  exercise.  For  MAPs  1  and  2,  the  Convoy 
Commanders  were  senior  Sergeants  (E-5),  and  for  MAP  3,  the  Convoy 
Commander  was  a  junior  Staff  Sergeant  (E-6). 

Most  of  the  Marines  who  participated  in  the  study  were  young  Lance 
Corporals  (E-3).  Excluding  the  aforementioned  Convoy  Commanders,  MAP  1 
had  three  noncommissioned  officers  (NCOs);  MAP  2  had  six  NCOs;  and  MAP  3 
had  six  NCOs.  The  average  age  of  the  participants  was  23.8  years.  Using 
Analsyis  of  Variance  (ANOVA)  and  paired  t-tests,  we  noted  that  there  was  a 
difference  in  age  between  MAPs  2  and  3  (p-value  =  0.0164)  with  MAP  2  having  a 
mean  age  of  22.75  years  and  MAP  3  having  a  mean  age  of  25.3  years.  This 
difference  arose  from  a  few  individuals  and  was  not  a  platoon-wide  trend.  While 
no  one  in  MAP  2  had  reached  their  30s,  MAP  3  had  a  31 -year-old  Staff  Sergeant 
and  a  35-year-old  Sergeant  who  swayed  the  average.  The  mean  time  in  service 
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for  all  participants  was  approximately  three-and-a-half  years  and  the  mean  time 
in  unit  was  approximately  two  years.  There  was  no  significant  difference  in 
platoons  for  these  attributes. 

2.  Apparatus 

The  project  used  twenty-six  Dell  Precision  M6300  laptops  from  a  suite  of 
the  Marine  Corps  Deployable  Virtual  Training  Environment  (DVTE)  package.  As 
in  previous  experiments,  computers  were  networked  using  D-LINK  DGS-2208  8 
port  switches.  Peripheral  equipment,  including  mice,  cables,  and  switches,  came 
from  the  standard  DVTE  package.  Unlike  Experiments  2  and  3,  the  network 
configuration  for  the  familiarization  training  was  the  same  as  for  the  training 
exercises.  We  networked  all  twenty-six  computers  together  using  seven 
switches,  allowing  all  machines  to  participate  together  in  the  same  environment. 

The  Weapons  Company  occupied  barracks  in  Camp  Pendleton  so  that 
they  had  access  to  a  parade  deck  and  large  classroom.  We  set  up  the  twenty-six 
computers  in  the  classroom,  networked  them,  and  left  them  configured  for  the 
week.  The  classroom  was  a  large  facility  configured  for  press  briefings  so  that 
we  had  easy  access  to  overhead  projection  and  power  at  individual  seats.  We 
used  the  back  six  rows  of  tables  in  the  room.  Each  table  row  had  eight  seats, 
and  we  used  the  seats  to  maximize  the  space  between  participants  as  much  as 
possible.  The  three  participants  in  a  vehicle  crew  sat  in  a  single  row,  with  two 
rows  having  two  vehicle  crews  (six  computers)  and  the  remaining  four  rows 
having  a  single  vehicle  crew  each.  For  the  majority  of  the  training,  the  trainer 
and  exercise  controller  each  had  a  computer  and  sat  facing  the  Marines 
conducting  the  exercise.  In  this  way,  both  the  trainer  and  the  controller  could 
monitor  the  virtual  action  on  the  screen  while  observing  the  action  of  the 
participants  in  the  room.  Figure  10  shows  the  room  configuration. 
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Figure  10.  Simulation  training  site  for  operational  unit  at  Camp  Pendleton,  CA 


As  in  Experiments  2  and  3,  participants  communicated  using  the  Joint 
Virtual  Tactical  Radio  (JVTR),  a  voiceover  IP  system.  From  the  lessons  learned 
in  Experiments  2  and  3,  we  attempted  to  reduce  the  complexity  of  the 
communications  problem  by  using  only  a  single  radio  net.  Only  the  vehicle 
commanders  used  the  radio  to  talk  on  a  single  net.  The  trainer  monitored  the 
platoon  traffic  using  his  computer  and  participated  as  higher  headquarters  on  the 
platoon’s  single  tactical  net.  Participating  Marines  were  taught  nothing  about  the 
radio’s  operation  except  how  to  push  a  key  to  talk. 

The  control  group  conducted  training  consisting  of  sand  table  exercises 
and  garrison  rehearsals.  This  platoon  used  the  ample  sandy  areas  outside  the 
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barracks  for  sand  table  exercises.  They  conducted  garrison  rehearsals  using  the 
unit’s  vehicle  on  the  parade  deck  outside  the  barracks.  The  platoon  commander 
controlled  the  exercises. 

As  mentioned  for  Experiments  2  and  3,  Marines  typically  operate  vehicles 
in  teams  of  three:  a  vehicle  commander  riding  in  the  passenger  seat,  a  driver, 
and  a  gunner  for  the  vehicle’s  weapon  system.  For  all  tactical  scenarios,  we 
grouped  participants  in  this  way.  Eight  vehicle  crews  participated  in  the  exercise. 
The  vehicles  were  all  HMMWVs  with  a  mix  of  0.50  caliber  machine  guns,  MK  19 
automatic  grenade  launchers,  and  TOW  missiles  that  reflected  how  the  MAPs 
trained  in  the  real  world  according  to  unit  rosters. 

As  in  Experiments  2  and  3,  we  used  statistical  group  comparison 
techniques  including  the  paired  t-test  for  analysis.  For  all  statistical  tests,  we 
used  an  alpha  level  of  0.05. 

3.  Procedure 

As  in  Experiment  3,  the  experiment  focused  on  training  four  tactical 
convoy  tasks:  reaction  to  an  lED,  reaction  to  ambush,  vehicle  recovery,  and 
casualty  evacuation.  We  chose  to  evaluate  the  effectiveness  of  the  training  in 
three  separate  domains.  First,  a  set  of  knowledge  tests  was  used  to  determine  if 
the  training  affected  participants’  academic  knowledge  of  the  skill  sets.  Second, 
surveys  were  used  to  determine  how  participants  assessed  their  own  proficiency 
and  their  unit’s  proficiency  in  each  skill  set.  Finally,  the  live  exercise  served  as 
an  indicator  of  the  participants’  practical  application  of  the  skills  in  the  real  world. 

We  used  the  same  knowledge  tests  that  had  been  used  in  Experiment  3 
with  Weapons  and  Field  Training  Battalion.  These  tests  consisted  of  eighteen 
multiple  choice  questions  each.  The  questions  on  the  two  tests  were  paired.  In 
this  way,  the  tests  were  different,  but  each  test  had  the  same  number  of 
questions  in  each  domain.  For  example,  if  Test  1  had  a  question  about 
unblocked  ambush.  Test  2  had  a  similar  question.  Data  from  Weapons  and  Field 
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Training  Battalion  was  used  to  ensure  the  tests  were  equally  difficult. 
Participants  took  one  of  the  tests  before  training  and  the  other  test  after  training. 
To  further  avoid  testing  bias,  half  the  participants  in  each  group  took  Test  1  first 
and  Test  2  second  and  vice  versa  for  the  other  half.  The  tests  are  included  in 
Appendices  P  and  Q 

The  survey  portion  of  the  experiment  also  used  the  same  paperwork  that 
was  devised  for  Experiments  2  and  3.  The  three-page  initial  demographic 
survey,  included  as  Appendix  M,  featured  several  questions  about  the  individual, 
such  as  age,  rank,  time  in  service,  computer  usage,  and  convoy  experience. 
Another  set  of  questions  intended  to  determine  the  participants’  attitudes  toward 
personal  computer-based  simulation  training.  The  next  set  of  questions 
determined  the  participants’  perception  of  tactical  skill  in  the  trained  skill  sets.  As 
an  addition  to  the  Experiment  3  survey,  a  follow-on  set  of  questions  determined 
the  participants’  perception  of  their  unit’s  tactical  skill  (that  is,  a  person  may 
believe  he  does  not  understand  how  to  react  to  an  I  ED  because  he  is  novice,  but 
that  the  platoon  as  a  whole  is  trained  in  that  skill).  Responses  to  all  attitude  and 
training  proficiency  questions  were  recorded  on  a  five-point  Likert  scale.  After 
the  training,  participants  completed  a  survey  with  the  same  simulation  attitude, 
individual  training  proficiency,  and  unit  training  proficiency  questions  that  were  on 
the  demographic  questionnaire.  They  also  had  a  page  to  record  subjective 
comments  as  answers  to  several  open-ended  questions  about  what  they  liked 
and  did  not  like  about  the  training.  The  questionnaires  were  the  same  for 
simulation  groups  and  for  the  control  group,  with  the  following  exceptions.  On 
the  post-training  questionnaire,  the  control  group  did  not  have  computer  attitude 
questions.  Also,  on  the  post-training  questionnaire,  the  simulation  groups 
answered  a  set  of  Likert  scale  response  questions  addressing  their  simulation 
experience.  The  post-training  survey  for  the  simulation  groups  is  included  as 
Appendix  N,  and  the  post-training  survey  for  the  control  group  is  included  as 
Appendix  O. 
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The  live  exercise  was  a  portion  of  a  regularly  conducted  training  package 
designed  for  reserve  infantry  battalions  training  at  Camp  Pendleton.  This 
battalion  was  the  sixth  to  complete  such  training.  The  training  package  consisted 
of  an  urban  patrolling  component  and  a  vehicle  convoy  component.  The  vehicle 
convoy  portion  consisted  of  two  days  of  repetitively  conducting  a  short  three-mile 
course  and  three  days  of  operating  on  a  longer  twenty-mile  course.  The  short 
course  included  the  following  evaluated  skill  sets:  establishment  of  a  snap 
vehicle  checkpoint  (VCP),  reaction  to  an  I  ED,  and  reaction  to  lED-triggered 
ambush.  Because  any  of  the  events  could  trigger  a  vehicle  or  personnel 
casualty,  the  exercise  adequately  served  as  a  mechanism  for  observing  the 
trained  skills  in  practice.  A  cadre  of  three  Marines  graded  the  short  convoy 
course.  All  of  the  training  cadre  had  graded  the  same  course  for  other 
companies  in  the  battalion.  Evaluators  graded  each  vehicle  on  the  course  using 
a  three-point  rating  of  untrained  (U),  trained  (T),  and  mastery  (M).  The  evaluator 
grading  system  was  developed  by  adding  the  three-point  grading  scale  to  the 
debrief  form  already  in  use  for  the  exercise,  and  the  resulting  evaluation  form  is 
included  as  Appendix  S.  All  evaluated  portions  of  the  course  were  positioned  in 
terrain  so  that  evaluators  could  observe  the  unit’s  activity  from  high  overwatch 
positions  or  from  within  the  convoy  itself. 

The  simulation  training  encompassed  the  two  separate  treatments  of  full 
mission  and  part-task  training.  VBS  2™  offers  several  terrain  databases,  and  we 
used  two  separate  databases  to  give  the  Marines  variety  in  their  training 
experience.  We  used  the  Twentynine  Palms  database  for  initial  training  including 
the  interface  familiarization  exercise.  The  Twentynine  Palms  database  is  a 
virtual  replication  of  the  Marine  Corps’  training  areas  at  the  Marine  Corps  Air 
Ground  Combat  Center  in  Twentynine  Palms,  California.  This  terrain  features 
open  desert  with  mountains  dividing  the  area  into  distinct  corridors.  The  terrain 
covers  a  large  space  more  than  60  kilometers  east  to  west  and  more  than  40 
kilometers  north  to  south.  For  the  more  difficult  training  exercises,  we  used  the 
Sahrani  database.  Sahrani  is  an  imaginary  island  developed  for  gamers  that 
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provides  a  Mediterranean  feel.  The  terrain  is  hilly  and  wooded  in  many  places. 
Rural  settlements  dot  the  landscape.  The  terrain  area  is  much  smaller,  featuring 
only  about  15  kilometers  at  its  widest  point  from  east  to  west;  the  whole  database 
covers  only  400  square  kilometers. 

Both  simulation  groups  started  training  by  completing  the  informed 
consent  paperwork,  demographic  survey,  and  initial  knowledge  test.  They  then 
completed  an  hour  of  interface  familiarity  training  based  on  the  work  from 
Experiment  1.  Afterward,  they  completed  a  short  exercise  called  “Training 
Wheels”  simply  to  familiarize  them  with  the  idea  of  working  together  in  a  vehicle 
convoy,  using  platoon  communications,  and  coordinating  movement  and  fires. 

After  the  initial  paperwork  and  training,  MAP  1  participated  in  the  full 
mission  training.  Throughout  the  four  days,  MAP  1  completed  five  convoy 
scenarios  in  the  Twentynine  Palms  database  and  two  scenarios  in  the  Sahrani 
database.  Each  of  the  scenarios  involved  all  four  skill  sets  in  question.  In 
general,  MAP  2  concentrated  on  casualty  evacuation  the  first  day,  vehicle 
evacuation  the  second  day,  reaction  to  ambush  the  third  day,  and  reaction  to  I  ED 
the  final  day.  The  final  exercises  did  not  involve  true  isolation  of  the  skill  sets;  for 
example,  participants  in  a  reaction  to  ambush  scenario  might  have  to  evacuate  a 
casualty  at  some  point  because  of  the  course  of  events.  Both  platoons  executed 
the  same  final  full  mission  exercise  in  Sahrani  terrain  at  the  end  of  the  training 
evolution.  Table  7  shows  the  details  of  the  training  plan  for  both  platoons. 
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Preparatory  training  for  aii 

Familiarity _ Interface  training  package  with  practice  exercise _ 

Training  Wheels  Convoy  movement  practice 

Fuii  mission  training  scenarios 

Level  1  Noble  Inert  lED,  Ambush  2x  fire  team  (FT),  Squad  with  inert  lED 

_ Pass _ and  single  vehicle  and  personnel  casualty _ 

Level  2  Gays  1x  FT,  1x  Squad,  1x  FT  with  daisy  chain  lED  and  vehicle 

_ Pass _ and  personnel  casualty,  civilian  farm _ 

Level  3  Rainbow  2x  trucks,  1x  Squad  with  lED,  civilian  farm,  1x  FT  with  daisy 

Canyon  chained  lED  and  vehicle  and  personnel  casualty,  1x  FT, 

_ civilians _ 

Level  4  Sahrani  1x  RPG  dismount  with  live  lED,  2x  RPG  dismounts,  1x  FT, 

Iguana  1x  Squad  with  daisy  chained  lED,  1x  RPG  dismount, 
_ intermittent  traffic  and  civilian  activity _ 

Part-task  training  scenarios 


Task 

Level  1 

Level  2 

Level  3 

Level  4 

Level  5 

Level  6 

Casualty 

Trng 

Noble  Pass 

Gays 

Cayo 

Hunapu 

Cresta 

Wheel 

Casualty 

Casualty 

Pass 

Casualty 

2x 

casualty 

Casualty 

2  sep  cas 
FT  /  RPG 

Cas  Pers 

3x 

casualty 

FT  +  Truck 

Free 

Play 

1  Truck 
Place 

cas 

Vehicle 

Trng 

Noble  Pass 

Rainbow 

Hunapu 

Cresta 

Hunapu 

Wheel 

Casualty 

Casualty 

Casualty 

2x 

casualty 

Cas  Veh 

2  sep  cas 

RPG/Tru 

ck 

Casualty 

2x 

casualty 

FT 

Free 

Play 

1  Truck 
Place 

cas 

Ambush 

Trng 

Noble  Pass 

Gays 

Hunapu 

Cayo 

Cresta 

Wheel 

Ambush 

Pass 

Ambush 

Ambush 

Free 

Ambush 

lx  FT  targ’t 

Ambush 
lx  FT 
inert 

2x  FT 

Live 

Squad+FT 

Play 

2x  Truck 

lED 

Trng 

Noble  Pass 

Rainbow 

Cayo 

Cresta 

Hunapu 

Wheel 

lED 

lED 

lED 

lED 

Free 

Casualty 

Undefended 

lx  lED 

RPG 

Team 

2  sep  lED 

RPG 

each 

lED+Chain 

RPG/2X 

FT 

Play 

2x  Truck 

Place 

lED 

Evaiuation  exercise  for  aii 

Sahrani  Ixel  1x  FT,  1x  FT  with  daisy  chained  lED  and  personnel  casualty, 

2x  trucks,  1x  Squad  with  daisy  chained  lED  and  personnel 
_ casualty,  1x  sniper,  intermittent  traffic  and  civilian  activity _ 

Table  7.  Convoy  training  matrix  for  simulation  groups 
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For  both  simulation  training  groups,  the  exercises  increased  in  complexity. 
We  varied  this  complexity  in  different  dimensions.  First,  the  exercises  had 
varying  degrees  of  assistance.  All  exercises  included  a  map,  and  that  map  may 
have  provided  a  location  and  description  of  the  problem  the  unit  would  react 
against,  just  a  location  of  the  problem,  or  no  information  about  the  problem  at  all. 
For  early  exercises,  an  overhead  view  of  the  exercise  area  was  projected  for  the 
participants  so  they  could  see  a  macro-level  view  of  their  actions  and  the  actions 
of  others.  Second,  the  exercises  had  varying  levels  of  collateral  activity.  Initial 
exercises  featured  nothing  other  than  the  exercise  problem  itself.  More  difficult 
levels  included  civilians,  cars  and  traffic,  animals,  and  buildings  and  built  up 
areas.  Third,  the  problem  difficulty  increased  within  each  skill  set.  More  difficult 
scenarios  involved  reacting  to  multiple  casualties  instead  of  just  one;  recovering 
multiple  vehicles  instead  of  just  one;  reacting  to  chained  lEDs,  fake  lEDs,  and 
remotely  detonated  lEDs;  and  complex  ambushes  with  integrated  Rocket 
Propelled  Grenade  (RPG)  gunners  and  potential  crossfire  situations.  Enemy 
activity  was  more  coordinated  in  advanced  exercises,  requiring  convoys  to  deal 
with  more  than  one  problem  at  once  (e.g.,  reacting  to  ambush  while  evacuating  a 
casualty).  Advanced  exercises  also  incorporated  some  enemy  indirect  fire. 

The  control  group  participated  in  exercises  that  mirrored  a  combination  of 
the  part  task  and  full  mission  groups’  activities.  To  mirror  the  part-task  activities, 
the  platoon  conducted  immediate  action  drills  with  actual  weapons  and  vehicles  on 
the  local  parade  deck.  Like  the  part-task  group,  the  control  group  concentrated  on 
a  single  skill  set  per  day.  To  mirror  the  full  mission  activities,  the  control  group 
conducted  sand  table  exercises  that  were  similar  to  the  full  mission  scenarios 
executed  in  simulation.  The  platoon  commander  used  the  full  mission  simulation 
materials  to  develop  his  sand  table  training. 

C.  RESULTS 

The  study  involved  three  different  measures  of  effectiveness:  knowledge 
tests  administered  before  and  after  training;  surveys  administered  before  and 
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after  training;  and  a  live  exercise  conducted  after  the  training.  Only  the  surveys 
produced  significant  results.  We  will  discuss  the  results  from  each  of  these 
evaluation  mechanisms  in  turn. 

1.  Knowledge  Tests 

We  graded  the  multiple  choice  tests  that  were  taken  before  and  after  the 
training  and  recorded  the  answers  in  a  spreadsheet  to  analyze  differences  in  the 
two  tests.  We  summarized  each  participant’s  performance  with  the  number  of 
correct  answers  produced  on  each  test.  We  then  looked  at  the  simple  difference 
between  correct  answers  on  the  before  test  versus  the  after  test.  We  used 
ANOVA  to  test  whether  there  was  a  difference  in  the  mean  performance  of  each 
platoon;  no  significant  difference  resulted.  We  also  conducted  t-tests  on  the 
performance  of  each  platoon  separately  to  test  the  hypothesis  that  participants’ 
academic  knowledge  of  convoy  operations  improved  after  training.  No  platoon 
performed  better  on  the  knowledge  test  with  significance. 

We  tested  to  see  if  the  tests  were  truly  equal  in  difficulty.  First,  we  looked 
at  participant  performance  on  each  18-question  test  regardless  of  whether  the 
participant  took  the  test  before  or  after  training.  On  average,  participants 
answered  9.87  questions  correctly  on  Test  1  and  9.09  questions  correctly  on 
Test  2.  Next  we  looked  at  participant  performance  when  taking  each  test  before 
training  and  then  after  training.  On  average,  participants  taking  Test  1  before 
training  answered  9.33  questions  correctly,  while  participants  taking  Test  1  after 
training  answered  9.86  questions  correctly.  Participants  taking  Test  2  before 
training  answered  an  average  of  9.18  questions  correctly,  while  participants 
taking  Test  2  after  training  answered  an  average  of  9.10  questions  correctly.  For 
both  ways  of  looking  at  the  issue,  there  was  no  significant  difference  in  the 
number  of  questions  answered  correctly,  so  no  evidence  suggested  a  significant 
difference  between  the  two  tests. 

Finally,  we  investigated  whether  participants  responded  to  the  simulation 
attitude  questions  based  on  any  specific  biases  from  the  demographic 

112 


information.  We  used  correlation  and  regression  analysis  to  check  each  of  the 
demographic  categories  and  found  no  relation  to  the  simulation  attitude  results. 
In  summary,  the  practical  application  training,  regardless  of  whether  it  was 
simulation,  traditional,  full  mission,  or  part  task,  resulted  in  no  change  in 
academic  performance  on  the  convoy  tests. 

2.  Surveys 

a.  Individual  Attitude  Toward  Simulation 

Participants  who  completed  the  simulation  training  took  surveys 
before  and  after  training  in  which  they  assessed  their  attitude  toward  simulation 
for  training.  They  answered  four  questions  using  a  five-point  Likert  scale  grading 
system.  We  used  this  construct  to  analyze  whether  their  attitude  toward 
simulation  changed  as  a  result  of  the  training  by  subtracting  the  difference 
between  the  before  and  after  ratings  for  each  skill.  We  analyzed  these 
differences  to  determine  whether  the  different  types  of  simulation  training 
produced  different  changes  between  the  two  platoons.  One  participant  in  the 
part-task  platoon  and  one  participant  in  the  full  mission  platoon  failed  to  complete 
the  full  week  of  training,  so  their  initial  surveys  were  not  considered  in  this  paired 
comparison.  In  total,  responses  were  analyzed  from  forty-six  participants. 

First,  we  combined  the  four  questions  into  a  single  composite  score 
by  averaging  the  answers  to  each  question.  We  used  a  t-test  to  test  the  null 
hypothesis  that  the  platoons  did  not  differ  in  their  change  in  attitude  toward 
simulation.  With  an  F-ratio  of  7.2142  and  a  p-value  of  0.0102,  we  found  that  the 
two  platoons  did  differ  with  the  full  mission  simulation  trained  platoon  having  a 
higher  assessed  attitude  than  the  part-task  simulation  trained  platoon.  With  this 
in  mind,  we  examined  more  closely  by  performing  the  same  analysis  on  each 
question  individually.  We  found  that  the  platoons  only  differed  for  two  of  the  four 
questions.  The  full  mission  simulation  trained  platoon  was  higher  for  “Computer- 
based  simulation  is  an  effective  training  tool”  and  “I  think  a  unit  should  use 
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computer-based  simulation  in  its  tactical  training.”  The  platoons  did  not  differ  for 
“Today’s  planned  training  will  improve  my  ability  to  conduct  convoy  operations” 
and  the  mirror  question  for  the  unit’s  ability.  Table  8  lists  the  data  for  this  analysis. 


# 

QUESTION 

F-RATIO 

P(>F) 

COMPOSITE  -  Attitude  toward  simulation 

7.2142 

0.0102 

1 

Computer-based  simulation  is  an  effective  training 
tool. 

6.7368 

0.0128 

2 

Today's  planned  training  will  improve  my  ability  to 
conduct  convoy  operations. 

2.0873 

0.1556 

3 

Today's  planned  training  will  improve  my  unit's  ability 
to  conduct  convoy  operations. 

1.2564 

0.2684 

4 

1  think  a  unit  should  use  computer-based  simulation  in 
its  tactical  training. 

8.6139 

0.0053 

Table  8.  Analysis  of  before  and  after  responses  to  statements  related  to  attitude 
toward  personal  computer-based  training;  yellow  highlight  indicates 
significant  difference  in  attitude  between  MAP  1  and  MAP  2 


Next,  we  analyzed  the  questions  by  platoon,  as  shown  in  Table  9, 
to  see  how  each  platoon’s  attitude  toward  simulation  changed  as  a  result  of  the 
exercise.  We  calculated  a  difference  between  Likert  scale  marks  for  each 
question  before  and  after  training.  We  then  used  a  t-test  to  investigate  the  null 
hypothesis  that  the  platoon’s  attitude  toward  simulation  did  not  change.  For  the 
full  mission  simulation  trained  platoon,  we  found  no  evidence  to  reject  the  null 
hypothesis.  However,  for  the  part-task  simulation-trained  platoon,  attitudes 
toward  simulation  went  down  for  all  but  “Today’s  planned  training  will  improve  my 
ability  to  conduct  convoy  operations.” 


Table  9. 


MAP  1 


MAP  2 


Question 


tTest  Prob>|t|  tTest  Prob>|t| 


Composite 

1.0675 

0.2973 

r 

1 

1.4316 

0.1663 

ma-:'- 

2 

1.0000 

0.3282 

-1.0446 

0.3075 

3 

-0.4393 

0.6647 

4 

1.2981 

0.2077 

Analysis  of  before  and  after  responses  to  statements  related  to  attitude 
toward  personal  computer-based  training  by  platoon;  red  highlight 
indicates  significant  decreased  attitude 
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In  summary,  the  two  simulation-trained  platoons  did  differ  in  how 
their  attitudes  toward  simulation  changed  as  a  result  of  the  training.  Specifically, 
they  differed  in  their  attitude  toward  simulation  in  general,  while  there  was  no 
difference  in  their  attitude  toward  the  training  they  received  in  this  particular  set  of 
training  sessions.  There  was  no  evidence  that  the  simulation  training  changed 
the  attitude  of  the  full  mission  simulation  trained  platoon,  but  the  part-task 
simulation  trained  platoon’s  attitude  toward  simulation  got  worse. 

b.  Individual  Tactical  Proficiency 

Participants  took  surveys  before  and  after  training.  On  these 
surveys,  they  assessed  their  proficiency  and  the  proficiency  of  their  unit  as  a 
whole  in  a  series  of  skills  using  a  five-point  Likert  scale  grading  system.  With  this 
construct,  we  analyzed  self-assessed  proficiency  by  subtracting  the  difference 
between  the  before  and  after  ratings  for  each  skill.  We  then  analyzed  these 
differences  to  determine  differences  between  platoons  and  perceived 
improvements  in  performance.  One  participant  in  the  part-task  platoon  and  one 
participant  in  the  full  mission  platoon  failed  to  complete  the  full  week  of  training, 
so  their  initial  surveys  were  not  considered  in  this  paired  comparison.  In  total, 
responses  were  analyzed  from  seventy  participants. 

We  combined  some  of  the  questions  to  create  composite  scores  by 
skill  group.  We  created  one  composite  score  encompassing  all  thirteen  tactical 
skills  as  a  group.  Additionally,  we  created  skill  group  composite  scores  including 
the  combination  of  two  questions  addressing  reaction  to  I  ED;  two  questions 
addressing  reaction  to  ambush;  three  questions  addressing  weapons 
employment;  and  three  questions  addressing  communications.  Vehicle  recovery 
and  casualty  evacuation  were  evaluated  as  stand-alone  questions.  We 
developed  the  composite  scores  for  the  skill  groups  by  averaging  the  responses 
from  the  questions  within  the  group. 

We  used  ANOVA  to  determine  whether  the  platoons  differed  in  their 
perceived  tactical  proficiency  before  and  after  training,  as  shown  in  Table  10.  We 
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found  that  the  platoons  did  differ  based  on  the  overall  composite  tactical  score  with 
an  F-ratio  of  7.95  and  a  p-value  of  0.0008.  Using  pairwise  t-test  comparisons,  we 
found  that  both  MAP  1  and  MAP  2  differed  from  MAP  3,  but  that  they  did  not  differ 
from  each  other.  That  is,  both  simulation-trained  groups  felt  that  their  proficiency 
increased  after  training,  but  the  traditionally  trained  group  did  not. 


# 

QUESTION 

F  RATIO 

P(>F) 

All 

COMPOSITE  INDIV- Tactical 

1/2  p  =  0.2347;  1/3  p  =  0.0002;  2/3  p  =  0.0098 

7.9489 

0.0008 

1-2 

COMPOSITE  INDIV  -  lED 

72  p  =  0.1386;  1/3  p  =  0.0002;  2/3  p  =  0.0149 

8.2399 

0.0006 

3-4 

COMPOSITE  INDIV -Ambush 

1/2  p  =  0.0243;  1/3  p  =  <0.0001;  2/3  p  =  0.0057 

13.4983 

<0.0001 

6-8 

COMPOSITE  INDIV -Weapons 

2.2811 

0.1101 

10- 

13 

COMPOSITE  INDIV  -  Communications 

2.2122 

0.1174 

Table  10.  Analysis  of  before  and  after  responses  to  statements  related  to 
composite  self-assessed  individual  proficiency  by  skill  set;  yellow 
highlight  indicates  overall  significant  difference  within  company,  bold 
indicates  between-platoon  significant  difference 

Next,  we  looked  at  the  skill  group  composite  scores  to  see  if  this 
trend  held  across  all  skill  sets.  We  used  the  same  analysis  methodology  of 
starting  with  ANOVA  to  determine  if  there  was  a  difference  between  platoons  and 
then  using  pairwise  t-tests  to  find  which  platoons  differed.  We  found  a  difference 
between  platoons  for  reaction  to  lED,  reaction  to  ambush,  vehicle  recovery,  and 
casualty  evacuation.  For  reaction  to  lED,  reaction  to  ambush,  and  casualty 
evacuation,  the  trend  from  the  overall  composite  score  was  reflected;  both 
simulation-trained  groups  felt  that  their  proficiency  increased  after  training, 
whereas  the  traditionally  trained  group  did  not.  For  vehicle  recovery,  only  the  full 
mission  simulation  trained  group  showed  a  significant  improvement  in  perceived 
proficiency  versus  the  traditionally  trained  group.  For  reaction  to  ambush,  all 
three  platoons  were  significantly  different  from  each  other;  this  was  the  only 
domain  in  which  the  two  simulation-trained  platoons  assessed  their  proficiency 
differently.  The  full  mission  simulation  trained  platoon’s  self-assessed  proficiency 
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increased  more  than  that  of  the  part-task  simulation  trained  platoon.  For 
weapons  employment  and  communications,  there  was  no  significant  different 
between  platoons  in  self-assessed  tactical  proficiency. 

We  examined  more  closely  the  tactical  skill  groups  that  showed 
significant  differences  by  platoon  to  see  if  the  individual  scores  reflected  the 
trends  of  the  parent  composite  grouping.  Table  11  lists  the  results.  For  reaction 
to  I  ED,  this  was  the  case.  Both  questions  showed  a  significant  difference 
between  platoons,  with  the  simulation-trained  platoons  having  a  higher  perceived 
proficiency  after  training  than  the  traditionally  trained  platoon.  The  reaction  to 
ambush  questions  differed,  however.  Both  questions  showed  a  difference 
between  platoons.  For  reaction  to  blocked  ambush,  the  full  mission  simulation 
platoon  had  higher  self-assessed  proficiency  than  the  part-task  simulation  trained 
platoon;  the  part-task  simulation  trained  platoon  did  not  differ  from  the 
traditionally  trained  platoon.  Data  analysis  for  the  reaction  to  the  unblocked 
ambush  question  followed  the  trend  of  the  overall  composite  score,  in  which  both 
simulation-trained  platoons  showed  significance  for  higher  self-assessed 
proficiency  than  the  traditionally  trained  platoon,  but  did  not  differ  from  each 
other. 
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# 

QUESTION 

F  RATIO 

P(>F) 

1 

INDIV  -  React  to  an  unexploded  lED 

1/2  p  =  0.0947;  1/3  p  =  0.0003;  2/3  p  =  0.0437 

7.1301 

0.0016 

2 

INDIV  -  React  to  an  lED  detonation 

1/2  p  =  0.3034;  1/3  p  =  0.0010;  2/3  p  =  0.0191 

6.2862 

0.0031 

3 

INDIV  -  Take  action  against  a  blocked  ambush 

1/2  p  =  0.0189;  1/3  p  =  <0.0001;  2/3  p  =  0.0910 

8.6607 

0.0005 

4 

INDIV  -  Take  action  against  an  unblocked  ambush 

1/2  p  =  0.1573;  1/3  p  =  <0.0001;  2/3  p  =  0.0017 

11.7701 

<0.0001 

5 

INDIV  -  Cordon  and  360  degree  security 

2.7234 

0.0730 

6 

INDIV  -  Employ  vehicle  machine  guns  /  weapons 

2.3307 

0.1051 

7 

INDIV  -  Mounted  fire  and  maneuver 

1.7102 

0.1886 

8 

INDIV  -  Shift  fires  /  cease  fires 

3.0269 

0.0551 

9 

INDIV  -  Vehicle  recovery  /  bump  plan 

1/2  p  =  0.4481;  1/3  p  =  0.0106;  2/3  p  =  0.0677 

3.6691 

0.0308 

10 

INDIV  -  Casualty  evacuation 

1/2  p  =  0.1756;  1/3  p  =  0.0366;  2/3  p  =  0.0008 

6.2918 

0.0031 

11 

INDIV  -  Communication  with  higher  headquarters 

1/2  p  =  0.1469;  1/3  p  =  0.0064;  2/3  p  =  0.1874 

3.9630 

0.0236 

12 

INDIV  -  Communication  between  vehicles  in  a 
convoy 

0.2459 

0.7827 

13 

INDIV  -  Communication  between  personnel  in 
vehicle 

0.7653 

0.4692 

Table  1 1 .  Analysis  of  before  and  after  responses  to  statements  related  to  self- 
assessed  individual  proficiency  by  individual  question;  yellow  highlight 
indicates  overall  significant  difference  within  company,  bold  indicates 
between-platoon  significant  difference 

In  summary,  we  saw  a  difference  between  platoons  in  tactical  skills 
overall.  In  general,  the  simulation-trained  platoons  did  not  differ  from  each  other, 
but  both  were  higher  than  the  traditionally  trained  platoon  for  self-assessed 
tactical  proficiency  improvement.  Reaction  to  blocked  ambush  did  not  follow  this 
trend,  because  the  two  simulation-trained  platoons  differed,  with  the  full  mission 
trained  platoon  having  the  higher  self-assessed  proficiency.  Casualty  evacuation 
did  not  follow  this  trend  either,  because  only  the  full  mission  simulation-trained 
platoon  showed  significant  increase  versus  the  traditionally  trained  platoon.  The 
weapons  and  employment  and  communications  skill  groups  showed  no 
difference  between  platoons. 
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Next,  we  looked  at  composite  scores,  and  then  each  question 
individually,  by  platoon  to  determine  whether  there  was  a  significant  difference  in 
self-assessed  proficiency  before  and  after  training.  We  took  the  difference  in 
Likert  scaled  responses  from  each  question  and  conducted  a  t-test  investigating 
the  null  hypothesis  that  there  was  no  change  in  self-assessed  proficiency. 
Across  all  three  platoons  and  all  questions,  there  was  no  decrease  in  self- 
assessed  proficiency;  participants  did  not  feel  they  got  worse  regardless  of 
platoon  or  skill  domain.  For  the  overall  tactical  composite  score,  both  of  the 
simulation-trained  platoons  increased  in  self-assessed  proficiency,  whereas 
analysis  showed  no  change  with  the  traditionally  trained  platoon.  All  platoons 
showed  no  difference  for  communication  between  vehicles  in  the  convoy  or 
communication  between  personnel  in  the  vehicle.  The  full  mission  simulation 
trained  platoon  felt  they  had  improved  with  significance  for  all  other  skill  sets;  the 
part-task  simulation  trained  platoon  agreed  for  all  skills  except  cordon  /  360 
degree  security  and  shift  /  cease  fires.  On  the  other  hand,  the  traditionally 
trained  platoon  did  not  show  any  change  in  self-assessed  proficiency  for  any 
skill.  That  is,  the  Marines  in  the  traditionally  trained  platoon  assessed  their 
individual  proficiency  for  all  analyzed  skills  the  same  regardless  of  whether  they 
had  had  the  traditional  training  or  not.  Table  12  summarizes  this  analysis. 
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Question 

MAP  1 

tTest  Prob>|t| 

MAP  2 

^  tTest  Prob>|t| 

MAP  3  L 

tTest  Prob>|t| 

Composite 

4.9092 

<0.0001 

3.3374 

0.0030 

-0.36678 

0.7171 

C  lED 

7.9530 

<0.0001 

3.2874 

0.0034 

0.1088 

0.9143 

C  AMB 

6.3471 

<0.0001 

3.3249 

0.0031 

-0.8614 

0.3979 

C  WEA 

2.8395 

0.0095 

1 .8433 

0.0788 

0.2413 

0.8115 

CCOM 

2.2170 

0.0373 

0.2277 

0.8220 

-0.4845 

0.6326 

PROF  1 

7.1895 

<0.0001 

2.5171 

0.0196 

0.0000 

1.0000 

PROF  2 

5.3911 

<0.0001 

3.8724 

0.0008 

0.1819 

0.8572 

PROF  3 

5.1627 

<0.0001 

2.0765 

0.0497 

-0.6174 

0.5430 

PROF  4 

6.0055 

<0.0001 

3.2137 

0.0040 

-1.0000 

0.3277 

PROFS 

2.5981 

0.0164 

1 .5554 

0.1348 

-0.5692 

0.5748 

PROF  6 

3.0262 

0.0062 

3.2137 

0.0040 

0.9010 

0.3769 

PROF  7 

2.1054 

0.0469 

2.4721 

0.0216 

0.0000 

1.0000 

PROFS 

2.4721 

0.0216 

-0.6171 

0.5435 

-0.3271 

0.7466 

PROF  9 

3.4583 

0.0022 

2.7324 

0.0122 

0.2531 

0.8024 

PROF  10 

2.5543 

0.0181 

4.7997 

<0.0001 

-0.5489 

0.5884 

PROF  11 

4.7500 

<0.0001 

2.6470 

0.0147 

0.3852 

0.7036 

PROF  12 

-0.1817 

0.8574 

-1.4168 

0.1705 

-1.2817 

0.2127 

PROF  13 

0.7676 

0.4509 

-0.6802 

0.5035 

-0.8106 

0.4259 

Table  12.  Analysis  of  before  and  after  responses  to  statements  related  to  self- 
assessed  individual  proficiency  by  platoon;  yellow  highlight  indicates 
significant  increased  proficiency 


c.  Unit  Tactical  Proficiency 

The  before  and  after  surveys  featured  the  same  tactical  skill  set  for 
unit  proficiency  as  for  individual  proficiency.  Thus,  participants  used  the  same 
five-point  Likert  scale  grading  system  discussed  above,  but  they  rated  their  unit’s 
proficiency  for  each  skill  instead  of  their  own.  An  example  demonstrates  the 
utility  of  this  evaluation  mechanism.  A  Marine  may  feel  that  he  is  quite  proficient 
at  reaction  to  ambush,  but  his  platoon’s  proficiency  does  not  match  his  own 
because  of  newly  joined  Marines,  unit  leadership,  or  some  other  reason. 
Because  the  survey  construct  matched  the  individual  survey  described  above, 
we  used  the  same  analysis  methodology. 

We  used  the  same  mechanism  of  creating  composite  questions 
and  breaking  the  questions  down  as  necessary  depending  on  significance  found. 
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That  is,  we  developed  an  overall  composite  tactical  skill,  composed  of  lED, 
ambush,  vehicle  recovery,  casualty  evacuation,  weapons  employment,  and 
communications  composite  skills.  We  used  ANOVA  to  test  the  null  hypothesis 
that  there  was  no  difference  by  platoon  in  ratings  for  the  overall  unit  tactical 
composite  score.  We  found  evidence  to  reject  this  hypothesis  with  an  F-ratio  of 
9.5772  and  a  p-value  of  0.0002.  Pairwise  t-tests  by  platoon  indicated  the  same 
trend  as  in  the  individual  composite  tactical  score;  that  is,  the  full  mission  and 
part-task  simulation  training  platoons  did  not  differ  significantly  from  each  other, 
but  they  both  differed  from  the  traditionally  trained  platoon.  In  all  cases,  the 
traditionally  trained  platoon’s  self-assessed  unit  proficiency  increased  less  than 
that  of  the  simulation-trained  platoons,  as  shown  in  Table  13. 


# 

QUESTION 

F  RATIO 

P(>F) 

All 

COMPOSITE  UNIT -Tactical 

1/2  p  =  0.5399;  1/3  p  =  0.0001;  2/3  p  =  0.0012 

9.5772 

0.0002 

1-2 

COMPOSITE  UNIT  -  lED 

1/2  p  =  0.3117;  1/3  p  =  <0.0001;  2/3  p  =  0.0004 

12.9869 

<0.0001 

3-4 

COMPOSITE  UNIT -Ambush 

1/2  p  =  0685;  1/3  p  =  <0.0001;  2/3  p  =  0.0153 

9.9240 

0.0002 

6-8 

COMPOSITE  UNIT -Weapons 

1/2  p  =  0.6791;  1/3  p  =  0.0053;  2/3  p  =  0169 

5.1195 

0.0093 

10- 

13 

COMPOSITE  UNIT  -Communications 

1/2  p  =  0.2732;  1/3  p  =  0.0070;  2/3  p  =  0.1090 

4.0401 

0.0233 

Table  13.  Analysis  of  before  and  after  responses  to  statements  related  to 

composite  self-assessed  unit  proficiency  by  skill  set;  yellow  highlight 
indicates  overall  significant  difference  within  company,  bold  indicates 
between-platoon  significant  difference 

As  in  the  individual  case,  we  investigated  the  composite  skill  groups 

within  the  overall  tactical  skill  set  to  see  if  all  skill  groups  followed  the  same  trend. 

We  found  that  the  trend  held  for  all  composite  skill  groups,  so  we  examined  to 

the  final  level  to  determine  whether  the  trend  extended  to  the  individual  questions 

within  each  skill  group.  For  lED,  ambush,  vehicle  recovery,  and  casualty 

evacuation,  the  trend  remained  the  same.  However,  the  trend  did  not  extend  to 

individual  questions  for  two  of  the  tactical  skill  groups.  Within  the  weapons 

employment  skill  group,  the  “Shift  fires  /  cease  fires”  skill  violated  the  trend.  In 
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this  case,  the  full  mission  and  part-task  simulation-trained  platoons  differed  from 
each  other,  but  the  part-task  simulation-trained  platoon  did  not  differ  from  the 
traditionally  trained  platoon.  Within  the  communications  composite  group,  only 
the  “Communications  with  higher  headquarters”  skill  followed  the  trend.  There 
was  no  significant  difference  between  platoon  for  “Communication  between 
vehicles  in  a  convoy”  and  “Communication  between  personnel  in  a  vehicle.”  This 
unit  rating  trend  for  the  unit  communications  composite  group  mirrored  the  trend 
for  the  individual  communications  group.  Table  14  shows  these  results. 


# 

QUESTION 

F  RATIO 

P(>F) 

1 

UNIT  -  React  to  an  unexploded  lED 

1/2  p  =  0.1531;  1/3  p  =  <0.0001;  2/3  p  =  0.0006 

14.1274 

<0.0001 

2 

UNIT  -  React  to  an  lED  detonation 

1/2  p  =0.5686;  1/3  p  =  0.0002;  2/3  p  =  0.0009 

10.0209 

0.0002 

3 

UNIT  -  Take  action  against  a  blocked  ambush 

1/2  p  =  0.0502;  1/3  p  =  0.0005;  2/3  p  =  0.0963 

7.0846 

0.0019 

4 

UNIT  -  Take  action  against  an  unblocked  ambush 

1/2  p  =  0.1470;  1/3  p  =  <0.0001;  2/3  p  =  0.0033 

11.1145 

<0.0001 

5 

UNIT  -  Cordon  and  360  degree  security 

1/2  p  =  0.5859;  1/3  p  =  0.0050;  2/3  p  =  0.0202 

5.0354 

0.0099 

6 

UNIT  -  Employ  vehicle  machine  guns  /  weapons 

1/2  p  =  0.3144;  1/3  p  =  0.0259;  2/3  p  =  0.0012 

6.1877 

0.0038 

7 

UNIT  -  Mounted  fire  and  maneuver 

1/2  p  =  0.8513;  1/3  p  =  0.0328;  2/3  p  =  0.0203 

3.6778 

0.0319 

8 

UNIT  -  Shift  fires  /  cease  fires 

1/2  p  =  0.0142;  1/3  p  =  0.0033;  2/3  p  =  0.6681 

5.2798 

0.0080 

9 

UNIT  -  Vehicle  recovery  /  bump  plan 

1/2  p  =  0.9619;  1/3  p  =  0.0250;  2/3  p  =  0.0259 

3.6621 

0.0323 

10 

UNIT  -  Casualty  evacuation 

1/2  p  =  0.2584;  1/3  p  =  0.0006;  2/3  p  =  <0.0001 

13.4421 

<0.0001 

11 

UNIT  -  Communication  with  higher  headquarters 

1/2  p  =  0.8775;  1/3  p  =  0.0005;  2/3  p  =  0.8775 

9.2914 

0.0003 

12 

UNIT  -  Communication  between  vehicles  in  a  convoy 

1.6562 

0.2004 

13 

UNIT  -  Communication  between  personnel  in  vehicle 

1.4421 

0.2454 

Table  14.  Analysis  of  before  and  after  responses  to  statements  related  to  self- 

assessed  unit  proficiency  by  individual  question;  yellow  highlight  indicates 
overall  significant  difference  within  company,  bold  indicates  between- 
platoon  significant  difference 
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Thus,  the  unit  tactical  skill  ratings  showed  similar  trends  to  the 
individual  skill  ratings.  There  was  a  significant  difference  in  overall  composite 
score.  The  difference  occurred  between  the  simulation-trained  platoons  and  the 
traditionally  trained  platoon  but  not  between  the  two  simulation-trained  platoons. 
This  trend  extended  through  most  of  the  composite  skill  groups  and  individual 
skills.  Notable  exceptions  included  weapons  employment  and  communications. 
For  communications,  the  trend  in  differences  only  extended  to  communication 
with  higher  headquarters  but  not  to  internal  unit  communications. 

As  in  the  individual  case,  we  examined  each  question  individually 
by  platoon  to  determine  whether  there  was  a  difference  in  self-assessed  unit 
proficiency  before  and  after  training.  We  used  the  same  methodology  of 
conducting  t-tests  on  the  differences  between  the  responses  to  test  the  null 
hypothesis  that  there  was  no  change  in  self-assessed  unit  proficiency.  Here,  it 
became  readily  apparent  that  the  results  did  not  match  the  trends  of  the 
individual  analysis;  the  overall  tactical  composite  rating  demonstrated  this,  as 
shown  in  Table  15.  Overall,  neither  simulation-trained  platoon  changed  in  self- 
assessed  unit  proficiency.  On  the  other  hand,  the  traditionally  trained  platoon 
showed  a  decrease  in  self-assessed  unit  proficiency  with  an  F  ratio  of  -3.1907 
and  p-value  of  0.0041.  That  is,  the  simulation-trained  platoons  felt  that  their 
unit’s  proficiency  was  the  same  regardless  of  the  training  whereas  the 
traditionally  trained  platoon  rated  their  unit’s  proficiency  worse  after  training.  For 
the  full  mission  simulation  trained  platoon,  self-assessed  unit  proficiency  only 
increased  for  the  lED  and  ambush  skills;  there  was  no  change  for  all  other  skills. 
The  part-task  simulation-trained  platoon  showed  an  increase  in  self-assessed 
unit  proficiency  for  one  lED  skill,  employing  vehicle  machine  guns,  and  casualty 
evacuation.  They  showed  a  decrease  in  self-assessed  unit  proficiency  for  shift 
fires  /  cease  fires  internal  communications  between  vehicles  and  personnel.  The 
part-task  simulation-trained  platoon  showed  no  change  for  all  other  skills.  The 
traditionally-trained  platoon  showed  a  decrease  in  self-assessed  unit  proficiency 
in  all  skills  except  reaction  to  blocked  ambush  and  vehicle  recovery. 
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Question 


MAP  1 

tTest  Prob>|t| 


MAP  2 

tTest  Prob>|t| 


MAP  3  L 
tTest  Prob>|t| 


Table  15.  Analysis  of  before  and  after  responses  to  statements  related  to  self- 

assessed  unit  proficiency  by  platoon;  yellow  highlight  indicates  significant 
increased  proficiency,  red  highlight  indicates  significant  decreased 

proficiency 


d.  Simulation  Group  Qualitative  Ratings 

Participants  in  both  simulation  groups  rated  nine  statements  about 
the  training  on  the  survey  taken  after  training.  We  designed  these  statements  to 
gain  insight  into  the  experience  of  simulation  training  itself.  We  used  one  way 
ANOVA  to  test  the  null  hypothesis  that  there  was  no  difference  between  platoons 
on  the  responses  to  these  statements.  We  failed  to  reject  the  null  hypothesis. 
We  analyzed  the  responses  to  determine  whether  any  individual  statements 
yielded  surprising  results.  Since  we  found  no  difference  between  platoons,  we 
conducted  this  analysis  in  the  aggregate  by  combining  the  responses  from  both 
groups.  In  general,  participants  responded  to  all  questions  with  average 
markings  near  the  middle  rating.  The  two  statements  that  a  satisfied  participant 
would  rate  lower  indeed  showed  lower  responses  than  the  other  seven 
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questions.  In  summary,  analysis  of  the  qualitative  ratings  showed  no  difference 
between  platoons  and  produced  no  interesting  information  worthy  of  report. 
Figure  1 1  shows  these  results. 


This  training  mission  was  successful 


During  this  exercise,  I  felt  like  my  actions  in  the 
virtual  environment  had  no  consequences 

During  this  exercise,  I  felt  like  I  was  playing  a  game 


During  this  exercise,  I  felt  like  I  was  conducting 
training 

During  this  exercise,  I  felt  like  I  was  part  of  a  group 
working  together 

During  this  exercise,  I  felt  isolated  from  others 


< 

< 

This  computer  simulation  provided  sufficient  audio  cues 
for  me  to  know  what  was  going  on 

This  computer  simulation  provided  sufficient  visual  cues 
for  me  to  know  what  was  going  on 

The  training  value  of  this  exercise  came  from  the 
debriefing  and  not  the  exercise  itself 


1  1.5  2  2.5  3  3.5  4  4.5  5 


Figure  1 1 .  Analysis  of  qualitative  ratings  of  simulation  training  experience 


e.  Comments 

Participants  had  ample  opportunity  to  provide  qualitative  feedback 
on  the  last  page  of  the  surveys  taken  after  the  training.  Several  questions  served 
to  guide  the  participants’  comments.  We  asked  participants  to  list  the  easiest 
and  most  difficult  task  for  each  of  individual,  vehicle  crew,  and  unit  performance. 
We  asked  what  the  participant  liked  least  and  most  and  then  provided  space  for 
free  comments.  We  examined  this  feedback  by  grouping  the  feedback 
comments  into  categories  within  each  feedback  question.  For  example,  some 
participants  responded  that  situational  awareness  was  most  difficult  for  them  as 
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an  individual,  so  we  grouped  these  responses  together.  While  this  informal 
methodology  provided  no  means  for  statistical  analysis,  it  did  allow  us  to  examine 
the  multitude  of  purely  qualitative  comments  for  trends. 


Also,  we  conducted  an  open  interview  session  with  each  simulation 
training  platoon  at  the  end  of  the  training.  We  used  a  standard  set  of  questions, 
which  included  the  following: 

•  What  did  you  like  about  the  training? 

•  What  did  you  not  like  about  the  training? 

•  Did  you  like  the  progression  of  the  scenarios? 

•  Would  you  rate  the  training  with  a  thumb  up  or  a  thumb 
down? 

When  we  combined  the  organized  qualitative  input  from  the  survey 
forms  with  the  group  comments  in  the  interview,  we  noticed  several  points  worth 
mention.  For  the  simulation-trained  platoons.  Marines  commented  on 
communications  more  than  any  other  topic  area,  and  the  comments  were  both 
positive  and  negative.  Throughout  all  of  the  simulation  training,  communications 
posed  the  most  technical  glitches,  so  this  problem  received  ample  critique.  Most 
Marines  had  an  opinion  about  how  participants  should  communicate  as  a  vehicle 
crew,  but  their  opinions  did  not  necessarily  agree.  Some  liked  using  the  voice 
communications  as  a  crew  because  it  was  more  like  the  real  vehicle.  Others 
would  have  preferred  to  use  the  computer  communications  with  a  headset  like  a 
vehicle  intercom.  Since  the  vehicle  commander  had  to  wear  a  headset  to  use 
the  convoy  communications  channel,  he  could  not  hear  voice  communications  as 
well  as  the  radio  communications,  contributing  to  this  issue.  On  the  other  hand, 
convoy  communications  received  many  positive  comments.  Many  felt  that  the 
simulation  exercise  had  helped  their  unit  improve  in  the  area  of  unit 
communications.  They  wanted  the  driver  to  be  able  to  hear  the  convoy  radio  net 
like  in  a  real  HMMWV.  In  general,  trainees  liked  the  challenge  and  training 
experience  of  the  convoy  radio  net,  but  did  not  like  the  vehicle  crew 
communication  or  technical  glitches  imposed  by  JVTR. 
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Both  simulation  groups  remarked  about  the  challenges  of 
situational  awareness.  Both  groups  felt  that  the  simulation  training  helped  their 
unit  in  this  area,  but  the  training  also  posed  some  artificial  challenges.  For 
example,  audible  feedback  such  as  explosions  and  gun  shots  come  from  the 
computer  speakers,  so  they  provide  no  directional  cues  like  in  the  real  world.  On 
the  battlefield,  reactions  to  explosions  occur  instinctually  because  a  person  can 
determine  relative  direction  and  distance,  but  a  laptop  computer  provides 
inadequate  cueing  for  this.  Also,  VBS  2™  does  not  provide  a  view  out  of  the 
driver’s  window,  so  the  driver  cannot  see  on  the  left  side  of  the  vehicle.  The 
vehicle  commander’s  view  is  very  limited  on  the  right  side  of  the  vehicle.  The 
result  is  that  the  vehicle  commander  and  driver  have  much  less  visibility  than  in 
the  real  world  and  must  depend  on  the  gunner  to  know  what  can  be  seen  from 
the  vehicle’s  point  of  view. 

Both  simulation  platoons  spent  much  time  in  the  interview  sessions 
discussing  the  keyboard  interface.  In  short,  the  majority  of  the  participants 
sought  a  console  game-type  controller  for  the  simulation  instead  of  a  keyboard 
interface.  They  felt  that  such  a  controller  would  be  more  intuitive  and  more 
familiar  from  other  leisure  gaming  activities. 

In  general,  simulation-trained  participants  liked  the  progression  of 
the  scenarios.  The  full  mission  training  platoon  felt  that  the  progression  of 
scenarios  started  and  ended  at  appropriate  levels.  The  part-task  training 
platoon,  on  the  other  hand,  felt  as  a  group  that  the  early  scenarios  were  too  easy. 
However,  the  convoy  commander  adamantly  disagreed  and  pointed  out  that  the 
easy  scenarios  were  very  useful  to  him  to  work  out  the  bugs  in  his  immediate  drill 
execution  before  facing  the  complications  of  the  more  advanced  scenarios.  Both 
simulation  platoons  viewed  the  Twentynine  Palms  terrain  as  lower  fidelity  and 
useful  for  introductory  training  while  the  Sahrani  terrain  was  considered  higher 
fidelity  and  more  useful  for  advanced  scenarios.  Figures  12  and  13  show  scenes 
from  the  two  terrain  databases. 
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Both  simulation  groups  commented  extensively  on  the  AAR  tool. 
Both  groups  believed  the  AAR  tool  was  critical  in  providing  a  big  picture  view  of 
what  happened  during  the  exercise.  They  liked  the  playback  capability  and  liked 
seeing  where  others  had  been  in  the  main  action  events  of  the  training.  Most 
participants  felt  like  they  had  very  little  understanding  of  the  enemy  activity  until 
seeing  the  AAR  material,  and  they  felt  the  AAR  tool  clarified  the  enemy  situation 
considerably.  Both  groups  felt  the  AAR  tool  was  one  of  the  most  valuable 
aspects  of  VBS  2™  training. 

Both  simulation  platoons  reported  that  the  training  helped  to 
develop  the  unit’s  immediate  action  drills.  Immediate  action  drills,  particularly 
reaction  to  ambush,  were  often  listed  as  easy  tasks  for  the  simulation  platoons. 
Both  platoons  often  listed  immediate  action  drills  as  one  of  the  primary  benefits  of 
the  simulation  training. 

The  traditional  training  platoon’s  feedback  tended  to  focus  more  on 
the  conduct  of  immediate  action  drills  than  the  feedback  of  the  simulation 
platoons.  The  traditional  training  platoon  rarely  mentioned  situation  awareness 
or  unit  communication  in  their  feedback.  Most  notable  in  the  traditional  training 
platoon’s  feedback  was  the  lack  of  trend  at  all.  Most  Marines  responded  to  the 
easiest  and  hardest  questions  with  some  aspect  of  an  immediate  action  drill,  but 
the  responses  were  not  particularly  consistent  in  any  way.  The  single  point  of 
consistency  in  the  feedback  was  that  the  training  took  a  long  time  for  what  was 
accomplished. 

One  common  point  among  all  three  training  groups  was  the  desire 
to  train  on  Mine  Resistant  Ambush  Protected  (MRAP)  vehicles.  At  the  time  of  the 
training,  all  MRAP  vehicles  were  deployed  to  combat  theater,  and  thus  none 
were  available  for  training.  VBS  2™  does  not  contain  MRAP  vehicles  as  it  is 
deployed  on  the  DVTE  suite.  However,  the  Joint  Training  Counter-Improvised 
Explosive  Device  Operations  Integration  Center  (JTCOIC)  has  used  VBS  2™  as 
a  platform  for  its  activities  and  has  developed  an  MRAP  vehicle  that  can  be 

loaded  easily  onto  any  VBS  2™  platform.  Given  the  impending  combat 
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deployment,  the  Marines’  desire  to  train  with  MRAP  vehicles  was 
understandable,  and  with  available  modification,  VBS  2™  could  have  satisfied 
the  need.  Currently,  the  Marine  Corps  does  not  allow  such  local  modification  to 
DVTE  suites,  relying  instead  on  centrally  provided  updates,  so  the  MRAP  from 
JTCOIC  could  not  be  used  in  this  training. 

Likewise,  we  conducted  an  open  interview  session  with  the  platoon 
commanders  who  served  as  trainers  for  the  three  groups.  The  platoon 
commander  who  conducted  the  traditional  training  had  used  VBS  2™  in  the  past, 
so  he  had  a  perspective  of  the  potential  of  the  simulation.  All  three  officers 
agreed  that  the  strength  of  the  simulation  lay  in  the  AAR  tool’s  ability  to  aid  the 
trainer  in  providing  feedback  to  the  training  audience.  Still,  all  three  officers 
demonstrated  skepticism  about  using  personal  computer-based  training.  They 
were  not  convinced  that  the  training  was  worth  the  technical  troubles  of  setting  it 
up.  Furthermore,  the  simulation  platoon  commanders  felt  that  the  view  available 
in  a  personal  computer  was  too  restrictive  for  infantry  training.  All  three  officers 
agreed  that  training  on  a  personal  computer  would  pay  off  on  deployment  at  sea, 
where  no  other  training  was  possible  and  time  was  readily  available  to  set  up  the 
system. 


f.  Live  Exercise 

The  company  participated  in  a  six-day  live  field  exercise  with  two 
primary  training  objectives;  urban  patrolling  and  mounted  patrolling  (Figure  14). 
Both  exercises  were  based  out  of  the  Kilo  2  combat  town  area  in  Camp 
Pendleton.  The  mounted  patrolling  training  package  contained  two  segments: 
an  initial  short  course  and  a  longer,  more  comprehensive  course.  Marines  ran 
the  exercises  at  the  platoon  level,  and  vehicle  constraints  resulted  in  each 
platoon  maneuvering  with  six  vehicles.  As  previously  discussed,  the  short 
convoy  course  met  our  needs  as  an  evaluation  mechanism  for  the  simulation  and 
traditional  preparatory  training.  This  course  consisted  of  a  roughly  three-mile 
loop  with  three  major  events.  During  the  exercise.  Marines  were  exposed  to  an 
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unexploded  lED  to  gauge  reaction  expertise.  They  established  and  conducted  a 
hasty  vehicle  checkpoint.  The  final  station  was  initiated  by  a  simulated  I  ED 
detonation,  and  then  a  two-man  oppositional  force  unit  ambushed  the  convoy. 


Figure  14.  Marines  operating  vehicle  check  point  in  live  convoy  exercise 

A  cadre  of  six  Marines  controlled  the  entire  operation,  including  the 
urban  patrolling.  A  colonel  served  as  the  exercise  controller  and  officer  in  charge 
and,  for  the  most  part,  remained  at  a  base  station  near  the  combat  town.  The 
controller  team  had  essentially  no  military  resources;  they  relied  on  personal 
vehicles  or  rides  in  the  participating  unit’s  vehicles  for  transportation  and  they 
relied  on  the  participating  unit’s  communications  assets.  Three  Marines  shared 
responsibility  for  evaluating  the  mounted  patrolling  courses,  but  all  three  Marines 
had  other  responsibilities  as  well.  Typically,  one  or  two  Marines  would  monitor 
the  short  convoy  course.  For  all  short-course  evolutions,  a  single  evaluator  rode 
in  one  of  the  participating  platoon’s  vehicles.  This  unit  evaluator  dismounted 
whenever  the  unit  stopped  and  monitored  the  action  from  whatever  best  vantage 
point  could  be  gained  on  the  ground.  During  movement,  the  unit  evaluator 
simply  rode  in  the  vehicle.  The  controller  cadre  had  the  option  to  augment  the 
unit  evaluator  with  a  second  Marine  using  overwatch  positions,  or  perches,  as 
shown  in  Figure  15.  For  each  of  the  three  stations,  the  terrain  provided  a 
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suitable  perch  to  view  the  participating  platoon’s  activity  from  an  overhead  view. 
Using  the  unit’s  Motorola  radios,  the  controller  cadre  had  sufficient 
communication  to  control  the  exercise  and  interact  to  provide  meaningful  insight 
into  what  was  happening  with  the  participating  platoon  on  the  ground. 


Figure  15.  Watching  a  convoy  movement  from  an  overwatch  position 

In  previous  exercises,  the  controller  cadre  used  a  standardized, 
custom-made  form  to  record  observations  and  provide  debriefing  feedback  to  the 
training  unit.  The  form  divided  the  exercise  into  its  primary  tasks  and  subtasks 
and  provided  lines  to  write  information  about  each.  In  order  to  collect  data,  we 
modified  the  existing  form  by  adding  squares  for  each  of  ten  vehicles.  The 
controllers  used  a  three-point  scale  to  grade  each  vehicle  on  each  of  the  tasks 
and  subtasks.  The  scale  included  named  markings  of  “untrained,”  “trained,”  and 
“mastery.”  Evaluators  placed  one  of  the  three  letters  U,  T,  or  M  in  the  boxes  to 
grade  each  vehicle  for  each  subtask.  If  the  exercise  was  evaluated  by  more  than 
one  Marine,  the  unit  controller  riding  in  one  of  the  training  unit’s  vehicles  served 
as  the  primary  controller  and  recorded  all  grades.  The  augmenting  controller  or 
controllers  provided  feedback  to  the  primary,  who  incorporated  the  information 
into  his  overall  grade. 
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Each  of  the  platoons  conducted  two  fully  evaluated  runs  of  the 
short  convoy  course.  With  virtually  no  exception,  the  evaluators  rated  the 
platoons  as  “trained”  for  each  of  the  tasks  and  subtasks  for  all  participating 
vehicle  crews.  The  overwhelming  number  of  “trained”  ratings  resulted  in  no 
significant  difference  by  platoon,  by  vehicle  crew,  or  by  task  category. 
Observational  comments  from  controller  cadre  indicated  agreement  that  there 
was  no  noticeable  difference  in  platoon  performance  on  the  short  convoy  course. 

g.  Notes  About  Experiment  Execution 

(1)  Controller  Effects.  For  the  simulation  training  platoons, 
we  gradually  and  deliberately  increased  the  involvement  of  the  platoon 
commanders  who  were  acting  as  trainers.  We  recognized  the  importance  of  the 
trainer  as  the  link  between  the  unit’s  training  objectives  and  the  simulation 
exercise.  We  built  all  the  scenarios  used  in  the  training  before  the  training  was 
conducted.  With  this  done,  trainers  could  involve  themselves  with  the  simulation 
in  two  ways:  alter  the  scenario  during  the  exercise  run  using  the  real-time  editor, 
and  run  the  after-action  tool  to  display  action  from  the  exercise  that  best 
supported  the  learning  points  they  wished  to  convey. 

The  concept  of  “building  scenarios”  can  have  different 
meanings  for  different  trainers  and  training  systems.  In  the  context  of  this 
project,  a  “built  scenario”  included  all  players  and  gear  in  an  initial  starting 
position,  all  enemy  in  initial  starting  positions,  and  moderate  amounts  of  objects 
that  amplified  the  scene.  Examples  of  amplifying  objects  included  civilians  along 
the  street,  civilian  vehicles  parked  in  driveways,  and  dirt  pile  cues  near  I  ED  sites. 
Even  a  well-built  scenario  does  not  run  by  itself.  The  controller  must  monitor  the 
enemy  to  ensure  their  actions  are  plausible,  and  advanced  scenarios  included 
traffic  and  moving  civilians  that  had  to  be  added  and  controlled  through 
waypoints  and  other  techniques.  Thus,  the  real-time  editor  in  VBS  2™  provided 
ample  opportunity  for  the  trainer  or  controller  to  affect  the  exercise  to  meet 
training  objectives.  As  the  controller,  we  did  all  this  work  without  the  assistance 
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of  the  trainer.  Throughout  the  course  of  the  week,  we  gradually  involved  the 
trainer  more  until  the  trainer  was  doing  most  of  the  facilitation  in  the  final 
exercises. 

Likewise,  the  after-action  tool  records  the  exercise  and 
allows  a  user  to  re-play  the  exercise  while  operating  a  camera  that  can  view  the 
exercise  from  any  angle.  Both  the  real-time  editor  and  the  after-action  tool  allow 
the  user  to  view  exercise  activity  from  a  two-dimensional  map  view  with  icon 
representations  of  players,  or  from  a  three-dimensional  world  view  with  avatar 
representations.  Between  the  two  views,  the  freedom  of  movement  throughout 
the  exercise  environment,  and  the  ability  to  re-play  segments  multiple  times,  the 
user  has  many  different  options  available  for  using  the  after-action  tool  to  make  a 
learning  point  clear.  It  is  conceivable  that  two  different  trainers  could  have 
completely  different  debriefs  based  on  the  same  scenario  by  supporting  different 
learning  points  with  various  camera  angles  and  world  views. 

As  we  increased  the  involvement  of  the  two  platoon 
commanders,  it  became  apparent  that  their  styles  of  involvement  differed.  The 
full  mission  simulation  training  platoon  commander  used  both  the  real-time  editor 
and  after-action  tool  aggressively  to  drive  home  the  learning  points  he 
considered  most  important.  At  times,  we  had  to  reign  in  his  real-time  editor 
involvement  in  order  to  keep  the  exercise  standardized  enough  to  maintain  the 
integrity  of  the  experiment.  The  full  mission  platoon  commander  readily  took 
charge  of  the  after-action  tool,  manipulating  the  camera  on  his  own  to  drive  home 
his  points  about  unit  performance.  On  the  other  hand,  the  part-task  simulation¬ 
training  platoon  commander  took  a  much  more  reserved  approach  to  both  tools, 
relying  more  on  the  controller  for  support.  The  part-task  platoon  commander  was 
noticeably  more  reluctant  to  assume  control  of  the  after-action  tool,  allowing  the 
controller  to  choose  the  playback  world  view  and  camera  position  and  then 
tailoring  his  comments  based  on  what  was  projected.  We  did  not  anticipate  this 
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sort  of  difference  between  trainers,  so  we  did  not  build  a  mechanism  into  the 
experiment  to  quantify  it.  Nevertheless,  the  difference  was  noticeable  throughout 
the  exercise  week. 

(2)  Communications.  We  sought  to  run  the  experiment 
using  VBS  2™  as  Marines  would  use  it  straight  out  of  the  boxes  of  the  DVTE 
suite  with  no  other  external  support.  With  this  in  mind,  we  relied  exclusively  on 
the  Joint  Virtual  Tactical  Radio  (JVTR)  for  communications.  JVTR  is  a  Single 
Channel  Ground  and  Airborne  Radio  System  (SINCGARS)  radio  simulator  used 
to  train  Marines  how  to  use  the  SINCGARS  radio.  This  simulator  can  be  used  as 
a  communications  platform  for  other  simulations  in  DVTE.  In  DVTE,  one 
facilitates  communication  with  VBS  2™  on  a  single  computer  by  running  JVTR 
simultaneously.  This  setup  works  well  enough,  but  one  must  understand  that  the 
communications  platform  is  an  external  add-on  and  not  an  internal  component  of 
VBS  2™.  In  this  exercise,  we  only  had  a  single  radio  net.  The  vehicle 
commanders  used  the  net  to  communicate  as  a  convoy  unit.  The  convoy 
commander  also  used  this  net  to  communicate  with  the  platoon  commander 
acting  as  trainer.  In  this  capacity,  the  trainer  served  as  a  higher  headquarters  to 
receive  battlefield  reporting  and  provide  headquarters  guidance.  The  controller 
also  used  this  net  to  provide  administrative  instructions,  as  necessary,  such  as 
exercise  start  and  end  information. 

We  had  many  technical  problems  with  communications, 
some  of  which  interfered  with  the  exercise  execution.  JVTR  simulates  a  dual 
radio  set  with  one  radio  going  to  one  earpiece  of  the  headset  and  the  second 
radio  sounding  in  the  other  earpiece.  Since  we  were  only  using  one  net,  we  used 
only  one  radio.  During  the  first  two  days,  all  of  the  JVTR  radios  would  stop 
functioning  at  some  point  in  the  day.  Through  trial  and  error,  we  determined  that 
we  could  fix  the  problem  by  programming  both  JVTR  radios  with  the  same  net 
frequency.  Once  VBS  2™  was  launched,  we  could  not  change  the  radios  without 
stopping  the  simulation.  This  created  a  situation  where  it  was  usually  better  to 
have  the  participant  continue  the  exercise  without  communications  rather  than 
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shut  the  simulation  down,  fix  the  radio,  and  then  try  to  get  the  participant  back 
into  the  play  of  the  exercise.  We  got  around  this  problem  by  operating  VBS  2™ 
in  Windowed  mode.  Eventually,  we  ran  all  JVTR  radios  with  the  same  frequency 
and  the  problem  went  away.  However,  the  continuous  communications  troubles 
in  the  first  two  days  caused  a  discernable  amount  of  distress  in  the  conduct  of 
the  exercise. 

We  also  found  that  the  JVTR  radios  would  periodically  shut 
down  for  the  busiest  users.  Quite  simply,  a  busy  JVTR  radio  saturated  the 
computer’s  buffer  until  the  application  locked  up.  We  noticed  that  this  problem 
occurred  mostly  with  the  convoy  commander,  so  we  solved  the  problem  by 
providing  communications  for  him  on  a  separate  computer.  The  convoy 
commanders  readily  adapted  to  the  slight  inconvenience  of  operating  with  two 
laptops.  We  drastically  reduced  JVTR  problems  in  this  way,  and  the  few 
problems  that  did  occur  for  the  convoy  commander  could  be  remedied  quickly. 

(3)  Technical  SNAFUs  (Situation  Normal;  All  Fouled  Up). 
We  sought  to  use  the  scripting  in  VBS  2™  to  facilitate  the  casualty  evacuation 
and  vehicle  recovery  training.  A  trigger  allows  scripting  on  the  fly  to  accomplish 
various  simple  tasks.  For  both  skills,  we  wanted  to  have  explicit  control  of  which 
individual  or  vehicle  was  a  casualty.  We  used  a  fake  I  ED  to  give  the  effect  of  an 
explosion.  A  fake  lED  produces  an  explosion  but  has  no  associated  damaging 
effect.  We  then  used  a  trigger  to  disable  a  certain  vehicle  or  cause  a  certain 
casualty.  Originally,  we  had  these  triggers  pre-built  in  the  scenario.  We  used  the 
scripting  reserved  word  “this”  in  the  trigger.  Unfortunately,  this  caused  the  effect 
to  be  applied  to  the  whole  unit  instead  of  the  single  entity  breaking  the  trigger 
bound.  For  example,  we  set  a  trigger  to  injure  the  driver  of  the  vehicle  that 
tripped  the  trigger,  but  all  of  the  drivers  in  the  convoy  ended  up  getting  injured  at 
once.  After  a  couple  exercises,  we  fixed  this  problem.  Since  the  part-task 
simulation-training  platoon  did  casualty  evacuation  and  vehicle  recovery  in  the 
first  two  days,  this  problem  affected  them  exclusively.  While  most  participants 
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understood  the  technical  problem  and  appreciated  it  as  a  humorous  mistake,  the 
part-task  trained  platoon  got  an  unfair  sampling  of  technical  glitches  up  front 
because  of  this  error. 

In  previous  studies,  we  had  noticed  a  computer  glitch  in 
which  the  vehicle  crew  representations  got  mixed  up  in  the  simulation.  In  a 
vehicle  crew,  a  gunner  would  be  unable  to  control  his  avatar  and  would 
continuously  get  thrown  out  of  the  vehicle  despite  repeated  attempts  to  get  back 
in.  The  vehicle’s  driver  would  get  thrown  out  of  the  vehicle  and  be  unable  to  get 
back  in.  The  driver  still  would  be  able  to  drive  the  vehicle  using  keyboard 
controls.  However,  the  driver  would  be  left  standing  on  the  ground  while  the 
vehicle  drove  away,  based  on  his  keyboard  input,  such  that  the  driver  could  not 
see  where  the  vehicle  was  going.  This  glitch  happened  so  rarely  that  we  could 
not  figure  out  how  to  fix  it  or  replicate  it  for  diagnosis.  Marines  often  have  the 
initiative  to  work  through  even  the  most  inconvenient  of  problems,  and  a  few 
vehicle  crews  managed  to  continue  with  the  driver  using  the  gunner’s  screen  to 
control  the  vehicle  and  the  gunner  repeatedly  getting  back  in  the  vehicle  every 
few  seconds.  We  eventually  found  that  we  could  quite  easily  remedy  this 
situation  by  changing  either  the  driver’s  or  the  gunner’s  computer  to  map  view 
and  then  pressing  “Esc”  to  get  back  into  the  world  view.  However,  through  our 
initial  lack  of  understanding  of  the  problem — and  their  gallant  initiative — a  few 
Marines  suffered  undue  frustration  because  of  this  problem. 

(4)  OPORD  (Operational  Order)  Delivery.  We  intended  to 
run  the  simulation  training  with  minimal  input  to  the  unit’s  training  implementation. 
We  wanted  to  remain  facilitators  while  the  unit  leadership  remained  trainers,  and 
we  sought  to  avoid  blurring  that  line,  if  possible.  We  provided  the  basic 
scenarios  and  map  materials  in  support  of  the  training,  but  we  did  not  provide 
operational  orders  that  specified  the  background  and  mission  of  the  exercise. 
We  relied  on  the  platoon  commanders  and  their  staff  for  this  function.  To  keep 
the  workload  minimal,  we  provided  an  OPORD  shell,  included  in  Appendix  R,  to 
help  their  development  efforts. 
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We  noticed  less  input  from  the  platoon  staffs  in  the  area  of 
OPORD  development  and  delivery  than  we  had  intended.  We  had  tried  to  allow 
the  platoon  to  use  the  blank  slate  of  the  scenario  at  hand  to  best  suit  unit  needs, 
but  instead  ended  up  with  a  rather  blank  slate  for  the  participants.  The  Marines 
relied  heavily  on  the  OPORD  shell  we  provided,  reading  from  it  almost  verbatim. 
Participants  did  not  add  in  detail  from  the  maps  and  terrain  to  tailor  to  the 
exercise  at  hand.  Moreover,  they  did  not  add  in  their  own  Standing  Operating 
Procedures  (SOPs)  and  custom  guidance.  Thus,  participants  operated 
throughout  the  exercise  based  mostly  on  the  provided  map.  Little  planning 
affected  the  conduct  of  the  exercise.  However,  the  full  mission  simulation¬ 
training  platoon  put  noticeably  more  thought  and  effort  into  OPORD  preparation 
and  delivery.  While  the  simulation  training  we  conducted  certainly  did  not  orient 
on  mission  planning  skills,  we  could  detect  the  differences  in  platoon  preparation 
levels  in  exercise  execution. 

D.  DISCUSSION 

1.  Hypotheses 

At  the  most  general  level,  this  study  allowed  us  to  answer  the  two  base 
questions  quite  simply.  Preliminary  training  with  VBS  2™  positively  impacts 
performance  in  a  platoon-level  tactical  scenario,  so  we  reject  the  first  null 
hypothesis.  Specifically,  the  simulation  training  resulted  in  no  difference  in 
academic  performance,  significant  difference  in  self-assessed  proficiency,  and  no 
difference  in  live  training  performance.  Additionally,  there  was  not  a  difference 
between  using  a  full  mission  approach  versus  a  part-task  approach  to  training 
when  using  VBS  2™  for  platoon-level  tactical  training,  so  we  fail  to  reject  the 
second  null  hypothesis.  This  conclusion  held  across  all  three  evaluation 
mechanisms. 

We  must  interpret  the  results  associated  with  the  first  hypothesis  carefully. 
Essentially,  Marines  felt  better  about  their  tactical  skill  but  failed  to  demonstrate 
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improvement  in  a  live  setting.  One  must  keep  in  mind  the  confounding  factors  of 
the  human  element  in  both  the  participation  and  evaluation  of  this  exercise.  With 
this  in  mind,  one  can  use  this  study  to  conclude  that  the  simulation  training  did  no 
harm.  That  is,  the  simulation  training  was  at  least  as  good  as  the  traditional 
training.  However,  a  conclusion  that  these  results  prove  that  simulation  training 
is  better  than  traditional  training  would,  in  our  opinion,  risk  extrapolating  the 
information  too  far. 

One  must  also  remember  that  self-assessed  proficiency  did  not  differ 
significantly  for  all  skill  sets.  Most  notably,  the  training  seemed  to  impact 
communications  less  than  other  skill  sets,  particularly  communications  at  the 
vehicle  crew  level.  At  the  individual  level,  the  training  did  not  impact  self- 
assessed  proficiency  for  weapons  skills.  It  is  also  important  to  note  that  not  all 
significant  differences  occurred  as  the  result  of  improvement.  For  self-assessed 
unit  proficiency,  virtually  all  skills  showed  a  significant  difference,  but  the 
difference  occurred  because  the  control  group’s  self-assessed  proficiency 
decreased  after  training.  This  result  may  not  suggest  as  much  about  the  virtual 
training  as  it  does  about  the  traditional  training.  It  may  be  that  the  traditional 
training  revealed  unit  training  deficiencies  that  never  got  corrected  during  the 
exercise. 

Further,  one  must  be  aware  of  the  limitations  of  the  study.  Between  the 
two  simulation  groups,  the  potential  for  a  trainer  effect  was  demonstrated  through 
the  different  techniques  of  using  the  real-time  editor  and  after-action  review  tool. 
Technical  glitches  in  both  the  communication  and  the  simulation  application  itself 
interfered  with  the  training  experience.  Finally,  the  evaluation  of  the  live  exercise 
was  not  robust  enough  to  highlight  fine  differences  in  performance.  While  these 
limitations  do  not  make  the  study  results  useless,  they  do  impact  the  extent  to 
which  the  results  can  be  generalized. 
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2.  Insights 


We  would  like  to  differentiate  this  study  from  previous  pilot  studies 
discussed  in  this  thesis.  The  previous  pilot  studies  involved  groups  of  people 
who  were  not,  in  any  way,  representative  of  the  unit  structure  involved  in  the 
training.  This  study,  on  the  other  hand,  involved  an  infantry  company  with  a  unit 
structure  established  specifically  for  motorized  patrolling  and  convoy  operations. 
While  it  is  inherently  obvious  that  this  difference  would  change  the  training 
experience,  it  is  worthy  of  mention.  The  simulation  training  worked  far  better  with 
an  intact  leadership  structure.  Participants  understood  who  was  in  charge,  basic 
standard  procedures,  and  individual  personalities  of  the  unit.  These  differences 
changed  the  face  of  the  training  completely.  For  the  type  of  convoy  training 
studied  in  these  experiments,  unit  integrity  is  a  key  factor.  Putting  individuals 
through  VBS  2™  training  does  not  necessarily  mean  a  unit  or  the  individuals 
themselves  are  trained  for  the  operation  at  hand.  Rather,  the  training  effect,  if 
there  is  to  be  any  at  all,  comes  from  training  the  unit  as  it  will  train  live  and  as  it 
will  fight  in  combat. 

This  project  also  highlighted  a  difference  in  the  training  audience  between 
the  reserve  infantry  company  and  the  active  duty  training  cadre  at  Weapons  and 
Field  Training  Battalion.  We  asked  for  a  show  of  hands  for  both  of  the  simulation 
platoons  if  the  Marines  were  college  students  immediately  prior  to  mobilization. 
Only  a  few  Marines  had  not  been  enrolled  full-time  in  college  immediately  prior  to 
deployment  to  Camp  Pendleton.  We  did  not  work  to  determine  the  courses 
these  Marines  had  been  taking  or  their  academic  goals  and  interests. 
Nevertheless,  the  mere  fact  of  their  almost-universal  college  enrollment 
separates  these  reservists  from  their  active  duty  counterparts  to  some  degree. 
The  assumption  that  the  findings  from  this  study  can  be  generalized  to  all 
Marines  is  confounded,  to  some  extent,  by  this  factor. 

Some  are  quick  to  tout  personal  computer-based  simulation  as  a  training 
solution  in  order  to  appease  the  technophilic  generation  of  young  adults  who  fill 


140 


the  junior  enlisted  ranks  of  the  Marine  Corps.  While  these  young  Marines 
typically  owned  computers  and  used  cell  phones,  they  were  not  necessarily 
gamers.  Moreover,  the  infantry  is  often  a  target  audience  for  personal  computer- 
based  simulation,  and  the  infantryman  may  be  less  technophilic  than  his 
counterparts  in  other  job  specialties.  While  both  groups  assessed  themselves  as 
more  proficient  in  the  selected  tactical  skills,  neither  group  appreciated  simulation 
training  more  after  the  exercise.  Moreover,  the  verbal  interview  after  the  exercise 
resulted  in  mixed  reactions  to  the  training.  Some  Marines  liked  the  training, 
some  did  not,  and  most  had  somewhat  ambivalent  views  toward  it.  Readily 
assuming  that  a  Marine  will  appreciate  simulation  training  simply  because  he 
uses  a  personal  data  device  and  the  World  Wide  Web  is  a  fallacy.  Marines  will 
appreciate  training  that  makes  them  better  warfighters  regardless  of  the  level  of 
technology  involved. 

Both  platoon  commanders  and  participants  lauded  the  after-action  review 
tool  as  critical  to  the  positive  VBS  2™  experience.  Likewise,  proactive  facilitation 
of  the  exercise  impacts  training.  Unit  leadership  can  be  involved  in  both  the  run¬ 
time  execution  of  the  exercise  and  the  debriefing  afterward.  Further,  VBS 
can  accommodate  different  leadership  styles  effectively.  However,  using  the 
real-time  editor  and  after-action  review  tools  effectively  involves  a  learning  curve. 
One  can  easily  overcome  this  with  a  few  exercise  runs,  but  one  cannot  assume 
that  unit  leaders  will  get  the  most  effect  out  of  the  simulation  training  based  only 
on  their  tactical  expertise.  Leaders  must  get  their  hands  dirty  with  the  simulation 
operation. 

As  we  increased  the  level  of  platoon  commander  involvement  in  the  use  of 
the  real-time  editor  and  after-action  review  tools,  we  noticed  subtle  differences  in 
their  use.  Typically,  the  platoon  commanders  used  the  third-person  world  view 
during  the  conduct  of  the  exercise.  Administratively,  we  typically  used  the  map 
view.  During  the  debriefing,  platoon  commanders  started  the  week  with  a 
preference  for  the  third-person  world  view  and  started  using  the  map  view  more 
as  the  week  went  on.  The  views  differ  in  the  level  of  detail  provided  to  the  eye. 
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The  third-person  world  view  makes  the  scene  look  realistic,  but  the  map  view 
shows  important  battle  information,  such  as  unit  location  and  weapon  orientation, 
explicitly.  We  found  that  most  debriefing  comments  that  were  supported  by  the 
after-action  review  tool  tended  to  involve  one  of  three  topics:  firing  unit’s  sight 
perspective,  weapon  orientation  and  shot  direction,  and  entity  position  relative  to 
other  entities.  The  third-person  world  view  is  more  suitable  to  see  a  situation 
from  a  gunner’s  point  of  view.  The  map  view  better  demonstrates  the  direction  of 
a  shot;  in  fact,  this  information  is  explicitly  displayed  as  a  line  on  the  map. 
Relative  position  is  often  more  easily  viewed  at  the  macro  level  with  the  map 
view,  but  in  close  terrain  with  hills  or  buildings,  the  third-person  world  view  is 
more  useful.  One  could  study  Best  Practices  for  the  use  of  the  different  tools  at 
length,  but  it  is  worthy  of  note  here  that  the  best  use  requires  practice  and 
experience.  Just  as  a  vehicle  driver  must  learn  the  simulation  interface  to  drive 
in  the  trainer  as  in  the  real  world,  the  trainer  must  learn  to  use  the  real-time  editor 
and  after-action  tools  to  train  effectively. 

Scenario  testing  is  critical  to  a  positive  training  experience.  An  exercise 
controller  can  easily  test  many  aspects  of  the  scenario  by  himself  while 
constructing  the  scenario.  VBS  2™  provides  a  very  convenient  way  to  test  run 
the  scenario  from  the  editor.  However,  this  practice  does  not  test  the  scenario 
with  group  interaction.  We  missed  the  technical  problem  of  all  drivers  becoming 
casualties  as  the  result  of  one  trigger  because  we  were  unable  to  test  the 
scenario  with  a  group.  A  single  developer  cannot  make  a  finished  training 
product;  the  developer  must  be  able  to  test  his  exercise  on  a  group  of  people. 
Fortunately,  a  controller  on  his  toes  can  correct  most  mistakes  in  stride  with  the 
real-time  editor,  but  a  unit  with  critical  dependence  on  a  given  scenario  should 
ensure  that  it  has  been  group  tested  before  execution. 

One  must  put  much  thought  into  exercise  control.  We  designed  all 
exercise  components  with  a  single  controller  in  mind  for  the  practical  reasons  of 
the  experiment.  We  relied  heavily  on  artificial  intelligence  driven  enemy, 
civilians,  and  traffic.  For  the  most  part,  we  made  all  of  these  entities  work 
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effectively  for  the  given  training  problem.  With  the  single  controller  in  mind,  we 
put  much  effort  into  scripting  these  entities  in  scenarios  beforehand.  We  found  in 
practice  that  it  was  easier  in  most  cases  to  create  entities  in  stride  using  the  real¬ 
time  editor.  This  is  not  to  say  that  a  controller  should  start  with  a  completely 
blank  screen;  the  decision  to  add  entities  before  exercise  versus  during  the 
execution  is  a  balancing  act  predicated  on  controller  experience  and  ability.  A 
more  complex  exercise  could  easily  overwhelm  a  single  controller.  One  must 
make  consideration  for  multiple  controllers  and  support  for  players  acting  as 
civilians  and  oppositional  forces. 

JVTR  is  a  workable  solution  for  exercise  communications,  but  it  may  not 
be  the  best  solution.  JVTR,  even  operating  at  its  best,  brings  the  possibility  of 
radio  crashes  due  to  buffer  overload.  At  its  worst,  JVTR  introduces 
communication  problems  that  cannot  be  fixed  during  exercise  execution.  The 
problems  are  more  difficult  to  diagnose  than  any  VBS  2™  issues.  By  the  very 
nature  of  communications,  JVTR  problems  frustrate  users  and  undermine  the 
effectiveness  of  the  simulation  as  a  situational-awareness  training  tool. 
Certainly,  a  better  solution  would  be  an  internal  communications  capability  in 
VBS  2™  itself.  In  the  absence  of  such  software,  a  unit  might  be  better  advised  to 
use  external  communications  systems.  This  consideration  is  noteworthy  when 
considering  the  out-of-the-box  Best  Practices  for  DVTE  users. 

3.  Summary 

Based  on  the  results  of  this  study,  we  conclude  that  preparatory  training 
with  VBS  2™  is  at  least  as  good  as  preparatory  training  by  traditional  methods 
such  as  sand  table  or  garrison  rehearsal.  We  conclude  that  the  simulation  can 
be  used  as  a  part-task  trainer  or  a  full  mission  training  device.  With  these 
thoughts  in  mind,  personal  computer-based  training  is  not  the  single  answer  to  a 
training  problem,  but  rather  another  capable  and  worthy  tool  for  the  trainer’s 
toolbox.  VBS  2™  does  what  it  was  designed  to  do;  that  is,  it  provides  the  trainer 
flexibility  for  resource  constrained  environments.  Additionally,  personal 
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computer-based  training  complements  traditional  training  well.  When  Marines 
tire  of  the  sand  table  and  parade  deck,  the  computer  provides  an  invitingly 
different  perspective  and  experience.  Thus,  the  simulation  provides  flexibility  to 
the  trainer.  Considered  by  the  dollar,  this  advantage  may  not  seem  worthwhile, 
but  when  the  real  world  equipment  is  in  transit  or  otherwise  unavailable,  the 
training  areas  are  booked,  and  only  a  few  days  remain  to  get  ready,  personal 
computer-based  training  can  be  an  attractive  asset. 
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V.  EXPERIMENT  IN  SUPPORT  OF  HYPOTHESIS  2 


A.  INTRODUCTION 

In  Experiment  4,  we  found  that  training  with  VBS  2™  produced  a  positive 
result  for  small  unit  tactical  convoy  training,  although  changes  in  performance 
were  limited  to  increases  in  self-assessed  proficiency.  With  knowledge  that 
personal  computer-based  training  was  beneficial,  we  sought  some  insight  into 
how  this  benefit  can  best  be  achieved.  We  based  our  work  on  careful 
contemplation  of  the  trainer  effect  we  had  noticed  in  Experiment  4.  That  is,  the 
full  mission  platoon  commander  engaged  himself  more  fully  in  the  conduct  and 
debriefing  of  the  simulation  exercise  than  the  part-task  platoon  commander  did. 
We  noticed  this  difference  despite  the  distinct  effort  to  involve  both  Marines.  This 
led  to  the  question  of  how  the  simulation  training  experience  would  differ  if  the 
direct  involvement  of  the  trainer  were  limited  rather  than  encouraged. 

We  felt  it  important  to  investigate  this  effect  because  of  the  unique 
potential  of  the  DVTE  concept.  Quite  commonly.  Marines  searching  for  VBS  2™ 
will  find  it  set  up  semi-permanently  or  permanently  in  local  simulation  centers. 
Our  personal  observation  confirms  this  is  true  across  major  Marine  Corps 
installations  such  as  MCAGCC  in  Twentynine  Palms,  California  and  the  I  MEF 
Simulation  Center  in  Camp  Pendleton,  California.  Units  have  even  pooled  their 
DVTE  resources  to  create  simulation  centers  such  as  the  3'^'^  Marine  Regiment 
simulation  center  in  Kaneohe  Bay,  Hawaii.  In  these  simulation  environments. 
Marine  unit  trainers  implement  the  training  exercise  through  the  support  of  a 
contractor  staff  who  facilitates  the  training.  Using  this  construct,  the  Marine 
trainer  provides  the  contractor  with  training  objectives  and  specific  desires  such 
as  terrain  or  equipment.  The  contractor  then  develops  the  appropriate  scenario 
and  controls  the  scenario  execution  for  the  unit. 
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However,  DVTE  is  different.  DVTE  puts  the  personal  computer-based 
simulation  in  the  hands  of  the  unit.  With  this  construct,  there  is  no  contractor 
support  staff.  Instead,  units  send  representatives,  typically  at  the  junior  enlisted 
level,  to  a  two-week  course  to  learn  how  to  operate  DVTE  as  a  suite  and  use 
each  of  the  suite's  simulation  applications.  The  DVTE  course  runs  on  a  regular 
basis  at  both  Camp  Lejeune  and  Camp  Pendleton  with  additional  courses  at 
other  installations  such  as  Twentynine  Palms.  Thus,  the  Marine  Corps  makes 
DVTE  training  available  to  all  units,  and  as  such,  a  unit  must  send  some 
representatives  to  DVTE  training  before  their  DVTE  suite  is  fielded.  These 
DVTE-trained  Marines  can  then  fill  the  role  of  the  contractor  support  in  the 
Marine  Corps'  various  simulation  centers.  That  is,  given  a  set  of  training 
objectives  and  exercise  specifications,  the  Marines  can  develop  training 
scenarios  and  operate  them  in  support  of  the  unit's  training  goals. 

Regardless  of  the  perceived  complexity  of  virtual  environments,  a  serious 
game  is  nothing  more  than  a  training  device,  a  tool  of  the  trainer.  With  this  in 
mind,  a  serious  look  at  training  effectiveness  is  more  than  an  investigation  of  the 
tool;  rather,  it  involves  investigation  of  the  trainer's  use  of  the  tool.  The  trainer's 
use  of  the  tool  involves  his  or  her  knowledge  of  the  simulation  as  well  as  what  he 
or  she  does  with  it.  A  simpler  example  illustrates  this  point.  A  hammer  is  a 
useful  tool,  and  with  it,  one  can  build  an  entire  building.  However,  without 
knowing  how  to  use  the  hammer,  one  cannot  build  anything  and  may  cause 
destruction  and  frustration  in  the  process  of  trying.  The  effectiveness  of  a 
serious  game  involves  a  relationship  between  the  simulation  and  the  trainer. 

In  either  of  the  DVTE  usage  situations  described  above,  a  third  party 
potentially  separates  the  trainer  from  direct  interaction  with  the  simulation 
exercise.  We  sought  to  investigate  the  potential  of  VBS  2™  if  this  separation 
was  removed.  We  have  discussed  the  concept  of  a  Marine  using  VBS  as  an 
extension  of  his  warfighting  skills,  and  this  idea  was  central  to  Experiment  4.  In 
Experiment  5,  we  investigated  the  concept  of  a  Marine  using  VBS  2™  as  an 
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extension  of  his  training  skills.  We  questioned  whether  the  trainer's  knowledge  of 
the  simulation  could  impact  the  simulation  experience. 

This  chapter  summarizes  Experiment  5,  in  which  we  tested  the  following 
hypothesis: 

•  A  trainer  who  has  been  trained  in  the  operation  of  the  simulation 
and  is  thereby  able  to  control  the  simulation  exercise  will  produce 
more  effective  training  than  a  trainer  who  effects  the  simulation 
exercise  through  a  third  party. 

B.  METHOD 

We  turned  to  a  school  environment  to  test  our  hypothesis.  The  school 
environment  provided  a  tested  structure  with  an  instructor  cadre  in  place.  Such  a 
situation  resulted  in  rigidly  defined  trainer  roles  that  reduced  the  potential  for  a 
confounding  trainer  effect  in  the  experiment.  We  selected  the  Enlisted 
Professional  Military  Education  (EPME)  program  at  Kaneohe  Bay,  Hawaii  for  our 
work.  EPME  Kaneohe  Bay  operates  a  seven-week  professional  training 
curriculum  commonly  known  as  the  Sergeant's  Course.  This  course  focuses  on 
the  junior  enlisted  leader  who  fits  the  profile  of  the  trainee  at  the  center  of  our 
previous  convoy  training  work,  so  the  products  we  had  developed  in  support  of 
the  first  thesis  hypothesis  naturally  fit  in  the  EPME  environment.  The  Sergeant's 
Course  offers  the  student  an  overview  of  a  broad  array  of  topics  spanning  the 
entire  Marine  Corps  from  close  order  drill  to  equal  opportunity.  Tactical  training 
only  receives  the  brief  overview  perspective  of  any  other  topic  area  with  most 
attention  paid  to  mission  planning  and  execution.  Class  sizes  usually  vary 
between  twenty  and  thirty  Marines.  The  instructional  package  follows  a  very 
structured  and  formalized  standard  managed  by  the  headquarters  at  Training 
and  Education  Command  (TECOM)  in  Quantico,  Virginia. 

EPME  Kaneohe  Bay  suited  our  experiment  well  because  the  unit  had  not 
yet  benefitted  from  its  parent  organization's  dissemination  of  VBS  2™  training 
packages  from  the  headquarters  in  Quantico.  The  school  had  a  full  DVTE  suite 
on  site  but  had  not  yet  opened  it  for  any  purpose  other  than  basic  inventory. 

147 


EPME  Kaneohe  Bay  was  ready  to  start  training  with  simulation,  though.  On  the 
school's  own  initiative,  the  instructor  cadre  used  white  space  in  the  course 
schedule  to  conduct  a  convoy  simulation  exercise  at  the  Virtual  Combat  Convoy 
Trainer  (VCCT).  For  a  half  or  full  day  of  each  course,  depending  on  schedule 
constraints,  EPME  students  would  run  standard  convoy  exercises  intended  to 
familiarize  them  with  the  execution  of  a  tactical  plan.  Since  students  hailed  from 
any  given  job  specialty  in  the  Marine  Corps,  the  school  did  not  expect  tactical 
prowess  and  was  not  particularly  concerned  with  learning  any  given  tactical  skill. 
Rather,  the  trainer  focused  on  the  troop  leading  procedures,  planning  steps,  and 
decision-making  skills  of  a  tactical  operation  and  used  the  convoy  exercise 
execution  as  a  forum  for  the  students  to  practice  these  concepts  in  an  actual 
mission. 

1.  Participants 

Participants  in  the  experiment  included  three  Sergeant’s  Course  classes 
at  EPME  Hawaii.  The  experiment  involved  seventy-nine  participants  altogether 
with  twenty-four  participants  in  the  first  class,  twenty-three  participants  in  the 
second  class,  and  thirty-two  participants  in  the  third  class.  All  participants  were 
Marine  Corps  Sergeants.  We  did  not  keep  demographic  statistics  for  the  first 
class.  For  the  second  and  third  classes,  the  Sergeants  had  an  average  of  5.81 
years  in  the  Marine  Corps  (standard  deviation  1.72  years).  Participants 
averaged  25.33  years  in  age  (standard  deviation  3.11  years).  Participants  came 
from  every  facet  of  the  Marine  Corps  with  fifty  different  job  specialties 
represented,  including  Marines  from  ground,  aviation,  and  logistics  elements. 
Eleven  Marines  participated  in  the  study  as  trainers,  although  only  five  Marines 
saw  the  study  from  beginning  to  end  due  to  personnel  turnover.  Trainers  ranged 
in  rank  from  Corporal  to  First  Sergeant,  averaged  28.50  years  in  age  (standard 
deviation  4.81  years)  and  8.98  years  in  service  (standard  deviation  4.44  years). 
Seven  of  the  trainers  had  infantry  backgrounds;  the  others  came  from  supporting 
job  specialties  throughout  the  Marine  Corps. 
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For  the  simulation-trained  groups,  we  collected  data  about  computer  and 
simulation  experience,  convoy  experience,  and  use  of  simulations  versus  sand 
tables  in  exercise  planning  and  rehearsal.  We  used  ANOVA  or  chi-squared  tests 
as  appropriate  to  test  the  null  hypothesis  that  the  groups  did  not  differ  any  of  the 
categories.  We  found  no  evidence  to  reject  the  null  hypothesis  for  any 
demographic  category. 

2.  Apparatus 

VCCT  provides  a  full  mock-up  of  military  vehicles  surrounded  by  large 
screen  projections  to  provide  a  fully  immersive  training  experience.  The  VCCT  at 
Kaneohe  Bay  contains  six  large  bays,  each  with  an  individual  replica  of  a 
HMMWV  or  MTVR.  Trainees  can  participate  in  the  exercise  seated  in  the 
various  seats  of  the  vehicle,  the  gunner's  station,  or  the  floor  area  around  the 
vehicle.  In  the  case  of  the  MTVR,  participants  can  sit  in  the  cargo  area  of  the 
vehicle  as  well.  Large  screens  around  the  bay,  combined  with  a  robust  sound 
system,  provide  fully  immersive  visual  and  audio  cues.  Typical  scenarios  involve 
skill  sets  examined  in  our  earlier  work,  including  reaction  to  lED,  reaction  to 
ambush,  casualty  evacuation,  and  vehicle  recovery.  A  contractor  staff  operates 
the  simulation  facility  with  most  activity  occurring  in  a  control  room,  providing  a 
standard  exercise  overview  from  an  array  of  computer  screens.  The  unit  trainer 
occupies  this  room,  providing  exercise  input  and  tracking  execution  details  with 
the  help  of  the  contractor  support. 

As  in  previous  experiments,  we  accomplished  VBS  2™  training  using  the 
DVTE  suite.  In  this  case,  we  used  the  freshly  unpacked  equipment  at  EPME 
Hawaii.  Specifically,  the  experiment  used  eighteen  Dell  Precision  M6300  laptops 
that  were  networked  using  D-LINK  DGS-2208  8  port  switches.  Peripheral 
equipment,  including  mice,  cables,  and  switches,  came  from  the  standard  DVTE 
package.  The  network  configuration  was  the  same  as  for  all  training  exercises, 
although  the  instructor  cadre  dismantled  the  equipment  and  re-packed  it  between 
different  segments  of  the  training.  We  networked  all  computers  together  using 
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switches,  allowing  all  machines  to  participate  together  in  the  same  environment. 
As  shown  in  Figure  16,  we  used  the  forty-seat  classroom  for  the  simulation 
training  with  trainees  occupying  the  first  two  rows  of  sixteen  seats  each. 


Figure  16.  EPME  instructors  training  a  Sergeant’s  Course  class  with  VBS  2™ 

at  Kaneohe  Bay 


Unlike  previous  experiments,  we  used  the  Combat  Net  Radio  (CNR)  in 
combination  with  the  internal  VBS  2™  communications.  CNR  is  a  proprietary 
software  package  designed  to  enable  communications  for  a  wide  variety  of 
simulations.  Caltryx,  the  company  who  developed  CNR,  has  cooperated  with 
Bohemia  Interactive  to  develop  a  CNR  interface  that  operates  with  the  VBS  2™ 
AAR  tool.  CNR  has  two  components:  CNR  SIM  serves  as  the  radio 
communications,  and  CNR  LOG  serves  as  the  recorder.  CNR  SIM  allows  users 
to  select  multiple  nets,  but  CNR  LOG  records  all  radio  traffic  on  the  network 

regardless  of  net.  For  this  reason,  we  only  used  CNR  for  convoy  net 
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communications.  For  communications  within  the  vehicle,  we  used  the  internal 
VBS  2™  vehicle  communications  feature.  This  feature  allows  players  to 
communicate  depending  on  their  vehicle  association  in  the  scenario. 

3.  Procedure 

We  designed  our  experiment  to  compare  three  treatments.  In  the  first 
treatment,  trainees  received  no  simulation  training  at  all.  This  control  group 
served  as  a  baseline.  In  the  second  treatment,  students  trained  in  the  VCCT  with 
the  instructor  cadre  administering  the  exercise  and  conducting  after  action 
reviews  through  the  third  party  support  of  the  simulation  facility  contractors.  In 
the  third  treatment,  we  trained  the  instructor  cadre  in  the  operation  of  VBS  2^''^. 
The  instructors  then  used  VBS  2™  on  their  own,  with  no  contractor  support,  to 
conduct  the  training  exercise.  For  convenience,  we  refer  to  these  treatment 
groups  as  the  control  group,  VCCT  group,  and  VBS  2™  group,  respectively. 
Because  of  the  small  class  size,  we  followed  three  separate  EPME  classes,  each 
class  serving  as  an  individual  group.  The  experimental  groups  went  in  the  order 
listed  above;  that  is,  the  first  class  was  the  control  group,  the  second  was  the 
VCCT  group,  and  the  final  class  was  the  VBS  2™  group. 

We  used  two  evaluation  mechanisms  to  determine  the  effectiveness  of  the 
training  for  each  group.  First,  we  developed  a  survey  mechanism  based  on  our 
products  for  Experiments  2,  3,  and  4.  Participants  filled  out  a  demographic 
survey  on  which  they  rated  their  self-assessed  skill  proficiency  and  attitude 
toward  simulation.  They  completed  a  post  training  survey  with  similar  self- 
assessed  skill  ratings  and  simulation  attitude  ratings.  They  also  answered  some 
questions  about  their  simulation  training  experience.  The  Experiment  5  surveys 
differed  from  the  previous  surveys  in  the  addition  of  skills  pertaining  to  troop 
leading  procedures,  mission  planning,  and  decision  making.  We  included  the 
demographic  survey  as  Appendix  T  and  the  post  training  survey  as  Appendix  V. 
Second,  we  used  a  knowledge  test.  We  used  the  knowledge  tests  developed  for 
Experiments  3  and  4  as  a  basis  for  the  Experiment  5  tests.  We  also  used  the 
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same  paired  test  design  discussed  in  the  previous  experiments.  That  is,  we  used 
two  different  tests,  each  with  twenty-four  multiple  choice  questions.  The 
questions  were  paired  between  tests.  For  example,  if  one  test  featured  a 
question  about  reaction  to  lED,  the  other  test  featured  a  different  question  with 
similar  intent.  Like  the  surveys,  the  Experiment  5  tests  differed  from  our  previous 
versions  because  they  contained  questions  related  to  troop  leading  procedures, 
mission  planning,  and  decision  making  to  reflect  the  goals  of  the  EPME  training. 
We  included  the  two  knowledge  tests  as  Appendices  Y  and  Z.  Since  the  control 
group  did  no  simulation  training,  they  only  took  the  knowledge  tests  as  a 
baseline,  answering  all  forty-eight  test  questions  in  one  composite  test.  The 
VCCT  and  VBS  2™  groups  took  the  survey  and  knowledge  test  before  and  after 
simulation  training. 

For  the  VBS  2™  group,  we  developed  a  training  package  for  the  EPME 
instructor  cadre  so  that  they  could  administer  simulation  training  to  the  EPME 
class.  We  sought  to  provide  a  turn-key  set  of  tools  that  the  trainers  could  use  at 
their  discretion  to  effect  the  course's  training  objectives.  We  based  the  package 
on  previously  developed  convoy  scenario  products.  Specifically,  we  created  a 
package  that  began  with  the  Familiarity  training  exercise  developed  in 
Experiment  1.  The  convoy  training  involved  a  modified  version  of  the  full  mission 
training  package  used  in  Experiment  4,  starting  with  the  Training  Wheels 
scenario  to  get  participants  working  together  as  a  convoy  team  and  then 
progressing  to  the  Noble  Pass  and  Rainbow  Canyon  scenarios  in  the  Twentynine 
Palms  terrain  and  the  Ixel  and  Iguana  scenarios  in  the  Sahrani  terrain.  In 
addition  to  the  scenarios  themselves,  the  training  package  included  the  hot  key 
cheat  sheet  developed  in  Experiments  1  and  2,  all  maps  and  the  OPORD  shell  to 
support  the  scenarios,  and  the  slide  show  we  used  to  support  the  Familiarity 
training  in  all  previous  experiments.  In  general,  we  gave  EPME  our  products 
from  the  Experiment  4  full  mission  training. 

We  conducted  the  instructor  cadre  training  during  a  down  period  between 
Sergeant's  Course  classes  over  a  three-day  period  (Figure  17).  Ten  instructors 
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participated  in  the  training,  although  ongoing  commitments  caused  a  few  of  the 
Marines  to  miss  portions  of  the  training.  We  started  the  first  day  of  training  by 
conducting  the  Familiarity  exercise.  We  did  not  run  the  exercise  exactly  like 
previous  experiments,  because  we  spent  extra  time  teaching  the  Marines  how  to 
run  the  Familiarity  training  by  themselves.  We  followed  the  Familiarity  training 
with  an  introduction  to  the  communications  systems  and  then  had  the  Marine 
staff  complete  the  Training  Wheels  scenario  as  a  single  convoy  unit.  Again,  we 
focused  on  showing  the  Marines  how  to  use  the  scenario  themselves  as  the 
trainer.  We  spent  the  latter  half  of  the  first  day  teaching  the  Marines  how  to  build 
a  scenario  by  going  through  the  full  menu  of  VBS  2™  entities  and  explaining 
each.  The  entire  training  exercise  was  practical  application  with  the  Marine  staff 
doing  each  trained  task  at  his  or  her  own  laptop  following  our  example  shown  by 
screen  projection.  On  the  second  day,  we  introduced  the  Marines  to  the  AAR 
tool,  showing  them  how  to  switch  views,  find  the  action,  and  use  the  tool  to  meet 
training  goals.  For  the  remainder  of  the  second  day,  we  went  through  each  of 
the  four  convoy  scenarios  in  detail.  Marines  took  turns  acting  as  the  controller 
while  the  rest  of  the  staff  participated  in  the  exercise.  We  provided  individual 
instruction  to  the  Marine  acting  as  controller  throughout  the  exercise,  focusing  on 
the  use  of  the  real  time  editor  to  use  triggers  to  achieve  battlefield  effects, 
provide  the  experience  of  civilians  in  the  environment,  and  manipulate  computer 
controlled  enemy  to  achieve  training  goals.  We  projected  the  controller's  screen 
on  the  Proxima  so  that  the  rest  of  the  class  could  see  what  the  controller  did  as 
we  introduced  new  real  time  editor  concepts.  Thus,  the  second  day  of  training 
was  a  combination  of  practical  application,  practice  as  a  user,  and  individualized 
instruction  oriented  on  giving  Marines  experience  as  an  exercise  controller. 
Throughout  the  training,  we  followed  a  list  of  notes  that  guided  our  instruction 
and  provided  the  Marines  an  outline  of  the  training  as  well  as  administrative 
information  such  as  file  locations  and  passwords.  We  included  this  notes  sheet 
as  Appendix  V. 
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Figure  17.  Instructor  training  with  VBS  2™  at  EPME  Kaneohe  Bay 


We  sought  to  investigate  the  effectiveness  of  the  train-the-trainer  package 
and  the  instructor  cadre's  impression  of  the  simulation  training.  We  used  a  pair 
of  surveys.  We  administered  the  demographic  survey  after  the  train-the-trainer 
instruction.  The  demographic  information  was  the  same  as  all  other  surveys 
used  in  previous  experiments.  As  in  the  other  surveys  oriented  on  simulation 
trainees,  we  asked  the  instructors  for  their  attitude  toward  simulation  training. 
We  asked  the  Marines  to  rate  how  effective  they  expected  VBS  2™  training  to  be 
for  each  of  the  tactical  skills  in  the  trainee  surveys.  We  also  asked  the  Marines 
to  rate  their  proficiency  in  a  variety  of  VBS  2™  controller  skills.  Finally,  we  asked 
for  subjective  comments  about  the  train-the-trainer  instruction.  The  post  training 
survey  asked  for  the  same  ratings  of  VBS  2™  effectiveness  to  train  the  various 
tactical  skills.  It  also  involved  the  same  self-assessed  proficiency  for  VBS  2™ 
controller  skills.  The  post  training  survey  asked  questions  about  the  simulation 
experience  from  the  controller  point  of  view.  Finally,  Marines  rated  which 
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simulation  trainer  they  would  prefer  for  each  of  the  tactical  skills,  choosing 
between  VCCT  and  VBS  2™.  The  survey  also  provided  the  opportunity  for 
subjective  comments.  We  included  the  demographic  survey  as  Appendix  U  and 
the  post  training  survey  as  Appendix  W.  As  in  previous  experiments,  all  survey 
products  for  Experiment  5  involved  questions  with  five-point  Likert  scaled 
responses. 

As  in  previous  experiments,  we  used  statistical  group  comparison 
techniques  including  the  paired  t-test  for  analysis.  For  all  statistical  tests,  we 
used  an  alpha  level  of  0.05. 

C.  RESULTS 

The  study  involved  multiple  measures  of  effectiveness.  The  knowledge 
tests  administered  before  and  after  the  virtual  training  provided  insight  into 
whether  participants’  factual  knowledge  of  the  skill  areas  changed  as  a  result  of 
training.  Participant  surveys  administered  before  and  after  training  measured 
changes  in  the  individuals’  self-assessed  proficiency,  providing  a  subjective  look 
at  changes  in  tactical  performance  as  a  result  of  the  training.  Comments  on  the 
participant  surveys  provided  further  subjective  insight  into  the  value  of  the 
training.  Finally,  instructor  surveys  administered  before  and  after  the  training 
provided  subjective  information  from  the  trainer’s  viewpoint.  We  will  discuss  the 
results  of  each  of  these  evaluation  mechanisms  in  turn  and  conclude  with  some 
notes  about  the  study  overall.  In  short,  none  of  the  evaluation  mechanisms 
produced  significant  results  in  the  data  analysis.  However,  observation  of  the 
exercise  in  progress  combined  with  analysis  of  the  subjective  comments 
provided  noteworthy  insights  about  the  impact  of  the  trainer  on  simulation 
training. 

1.  Knowledge  Tests 

This  study  involved  three  groups:  the  control  group  who  received  no 
virtual  training;  the  VCCT  group  in  which  trainers  implemented  training  through  a 
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third  party;  and  the  VBS  2™  group  in  which  trainers  implemented  training  directly 
by  administering  the  simulation  training  themselves.  The  control  group  took  a 
knowledge  test  with  no  virtual  training  at  all  in  order  to  establish  a  baseline.  The 
two  treatment  groups  took  knowledge  tests  before  and  after  their  virtual  training. 
We  graded  all  of  the  multiple  choice  tests  and  recorded  the  results  in  a 
spreadsheet  for  analysis,  paying  particular  attention  to  the  average  number  of 
questions  missed  overall. 

We  started  by  comparing  the  percentage  of  questions  missed  for  each 
individual  by  treatment  group.  We  used  ANOVA  to  test  whether  there  was  a 
difference  in  performance  on  the  final  test  by  group,  and  we  found  no  significant 
difference  (F  ratio  of  0.2194,  p-value  of  0.8035).  We  also  investigated  whether 
the  knowledge  of  either  simulation  trained  group  changed  after  training,  using  a  t- 
test.  For  both  groups,  no  significant  difference  in  the  number  of  correct  answers 
resulted  (for  the  VCCT  group,  t  Ratio  of  0.3184  and  p-value  of  0.7517;  for  the 
VBS  2™  group,  t  Ratio  of -0.6644  and  p-value  of  0.5089). 

The  control  group’s  knowledge  test  consisted  of  fifty  multiple  choice 
questions.  We  dropped  two  questions  and  split  the  test  in  half  to  make  two  tests 
of  twenty-four  questions  each  for  the  two  treatment  groups.  The  VCCT  group 
took  the  two  tests  as  planned  with  half  taking  Test  A  first  and  the  other  half  taking 
Test  B  first.  However,  an  administrative  error  resulted  in  the  VBS  2™  group 
taking  the  fifty-question  test  instead  of  Test  A.  That  is,  the  VBS  2™  group  either 
took  Test  B  or  the  entire  fifty-question  test  (which  included  the  twenty-four 
questions  from  Test  B).  We  corrected  this  problem  in  the  data  analysis  by  only 
grading  those  questions  that  were  on  Test  A.  However,  participants  from  the 
VBS  2™  group  who  took  the  fifty-question  test  before  training  saw  the  Test  B 
questions  twice. 
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2.  Participant  Surveys 


We  used  the  same  techniques  to  analyze  the  participant  self-assessed 
proficiency  that  we  used  in  Experiment  4.  Like  the  previous  experiment,  we 
analyzed  individual  attitude  toward  simulation,  individual  tactical  proficiency,  unit 
tactical  proficiency,  simulation  group  qualitative  ratings,  and  comments.  In 
general,  responses  from  the  VCCT  group  reflected  a  positive  training  experience 
and  improved  self-assessed  proficiency  while  the  VBS  2™  group’s  responses 
reflected  much  less  deviation  from  the  status  quo.  In  total,  we  analyzed 
responses  from  twenty-three  participants  in  the  VCCT  group  and  thirty-two 
participants  in  the  VBS  2™  group. 

a.  Individual  Attitude  Toward  Simulation 

As  in  Experiment  4,  we  compared  responses  to  five-point  Likert 
scaled  questions  from  surveys  taken  before  and  after  the  VCCT  and  VBS 
training.  We  started  by  creating  a  composite  score  of  the  four  simulation  attitude 
questions  by  averaging  the  answers.  We  used  a  t-test  to  test  the  null  hypothesis 
that  the  platoons  did  not  differ  in  their  change  in  attitude  toward  simulation.  With 
an  F-ratio  of  4.1614  and  a  p-value  of  0.0465,  we  rejected  the  null  hypothesis  and 
concluded  that  the  two  treatments  differed  with  the  VCCT  group  having  a  higher 
assessed  attitude  than  the  VBS  2™  group.  We  scrutinized  the  individual 
questions  to  see  if  this  difference  was  reflected  across  all  questions.  The  trend 
held  for  all  questions  except  “I  think  a  unit  should  use  computer-based  simulation 
in  its  tactical  training.”  For  this  item,  the  responses  reflected  no  significant 
difference.  Table  16  shows  this  analysis. 
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# 

QUESTION 

F-RATIO 

P(>F) 

COMPOSITE  -  Attitude  toward  simulation 

16.8145 

0.0001 

1 

Computer-based  simulation  is  an  effective  training 
tool. 

4.1614 

0.0465 

2 

Today's  planned  training  will  improve  my  ability  to 
conduct  convoy  operations. 

17.3745 

0.0001 

3 

Today's  planned  training  will  improve  my  unit's  ability 
to  conduct  convoy  operations. 

23.5620 

<0.0001 

4 

1  think  a  unit  should  use  computer-based  simulation 
in  its  tactical  training. 

3.1431 

0.0822 

Table  16.  Analysis  of  before  and  after  responses  to  statements  related  to  attitude 
toward  personal  computer-based  training;  yellow  highlight  indicates 
significant  difference  in  attitude  between  VCCT  and  VBS  2™  groups 


Next,  we  analyzed  the  questions  by  group  to  see  how  each 
treatment  group’s  attitude  toward  simulation  changed  as  a  result  of  the  training 
(Table  17).  After  calculating  the  differences  between  before  and  after  responses, 
we  used  a  t-test  to  test  the  hypothesis  that  the  group’s  attitude  toward  simulation 
did  not  change.  For  the  VCCT  group,  we  found  evidence  to  reject  the  null 
hypothesis  and  determined  that  the  group’s  cumulative  attitude  toward  simulation 
improved.  Individual  scrutiny  of  each  question  reflected  this  trend  with  the 
exception  of  “Today’s  planned  training  will  improve  my  ability  to  conduct  convoy 
operations.”  On  the  other  hand,  while  the  VBS  2™  group’s  cumulative  attitude 
toward  simulation  reflected  no  significant  difference,  the  two  questions 
specifically  to  “today’s  training”  showed  a  significant  decline  in  attitude. 


Table  17.  Analysis  of  before  and  after  responses  to  statements  related  to  attitude 
toward  personal  computer-based  training  by  virtual  training  group; 
yellow  highlight  indicates  significant  increased  attitude,  red  highlight 
indicates  significant  decreased  attitude 
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In  summary,  the  attitudes  of  the  two  simulation  groups  differed. 
Both  groups  showed  an  increased  inclination  to  believe  that  a  unit  should  use 
simulation  in  its  tactical  training.  However,  when  evaluating  the  impact  of 
“today’s  training,”  the  VCCT  group  believed  more  strongly  after  the  training  that 
the  training  helped  at  both  individual  and  unit  levels  while  the  VBS  2™  group 
rated  these  questions  lower.  Additionally,  the  VCCT  group’s  belief  that 
simulation  is  an  effective  training  tool  increased  after  the  training  while  the  VBS 
2™  group’s  opinion  remained  the  same. 

b.  Individual  Tactical  Proficiency 

As  in  Experiment  4,  participants  rated  their  proficiency  in  various 
skills  before  and  after  training,  and  we  compared  the  responses  to  determine  if 
the  training  positively  or  negatively  impacted  their  opinion  of  their  skills.  As 
shown  in  Table  18,  we  started  with  composite  scores  and  broke  down  the  scores 
to  the  individual  question  level.  Analysis  began  with  ANOVA  to  test  the  null 
hypothesis  that  the  changes  in  self-assessed  proficiency  did  not  differ  between 
the  two  simulation  groups.  The  overall  composite  score  for  all  skills  provided 
evidence  to  reject  the  null  hypothesis  and  conclude  that  the  groups  differed. 
Specifically,  the  VCCT  group’s  self-assessed  proficiency  increased  more  than 
that  of  the  VBS  2™  group.  Breaking  the  analysis  down  by  category  and  then  to 
the  individual  skill  level  demonstrated  that  this  trend  extended  through  all  skills 
with  the  exception  of  “Shift  fires  /  cease  fires.” 
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# 

QUESTION 

F  RATIO 

P(>F) 

All 

COMPOSITE  INDIV- Tactical 

25.2534 

<0.0001 

1 

COMPOSITE  INDIV  -  lED 

13.3330 

0.0006 

1 

INDIV  -  React  to  an  unexploded  Improvised 
Explosive  Device 

14.7006 

0.0003 

2 

INDIV  -  React  to  an  Improvised  Explosive  Device 
detonation 

11.2612 

0.0015 

1 

COMPOSITE  INDIV -Ambush 

7.4722 

0.0085 

3 

INDIV  -  Take  immediate  action  against  a  blocked 
ambush 

8.9945 

0.0041 

4 

INDIV  -  Take  immediate  action  against  an  unblocked 
ambush 

4.1814 

0.0459 

1 

5 

INDIV  -  Cordon  and  360  degree  security 

8.8151 

0.0045 

1 

COMPOSITE  INDIV -Weapons 

10.4512 

0.0022 

6 

INDIV  -  Employ  vehicle  machine  guns  /  weapons 

14.3834 

0.0004 

7 

INDIV  -  Mounted  fire  and  maneuver 

8.4603 

0.0053 

8 

INDIV  -  Shift  fires  /  cease  fires 

3.5529 

0.0651 

1 

9 

INDIV  -  Vehicle  recovery  /  bump  plan 

9.9786 

0.0027 

1 

10 

INDIV  -  Casualty  evacuation 

11.8349 

0.0012 

1 

COMPOSITE  INDIV  -  Communications 

34.6655 

<0.0001 

11 

INDIV  -  Communication  with  higher  headquarters 

22.6119 

<0.0001 

12 

INDIV  -  Communication  between  vehicles  in  a 
convoy 

24.7351 

<0.0001 

13 

INDIV  -  Communication  between  personnel  in 
vehicle 

26.2611 

<0.0001 

1 

COMPOSITE  INDIV  -  Preparations 

19.9135 

<0.0001 

14 

INDIV  -  Execute  the  troop  leading  steps  using 
BAMCIS 

10.1581 

0.0024 

15 

INDIV  -  Conduct  mission  analysis  using  METT-T 

15.1204 

0.0003 

16 

INDIV  -  Receive  a  5  paragraph  operational  order 

8.8146 

0.0045 

17 

INDIV  -  Give  a  5  paragraph  operational  order 

16.0316 

0.0002 

Table  18.  Analysis  of  before  and  after  responses  to  statements  related  to  self- 
assessed  individual  proficiency;  yellow  highlight  indicates  significant 
difference  between  simulation  training  groups  with  VCCT  group  having  the 

higher  ratings 
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Next,  we  calculated  the  difference  between  before  and  after  self- 
assessed  proficiency  ratings  for  each  individual  and  conducted  t-tests  by  group 
to  test  the  null  hypothesis  that  each  group’s  proficiency  rating  did  not  change 
(Table  19).  For  the  VCCT  group,  we  found  evidence  to  reject  the  null  hypothesis 
with  self-assessed  proficiency  showing  significant  increase  for  all  categories  and 
individual  skills.  On  the  other  hand,  for  the  VBS  2™  group,  only  two  categories 
showed  significant  difference:  both  the  ambush  and  weapons  skills 

demonstrated  increase  in  self-assessed  proficiency.  Ratings  showed  no 
significant  difference  for  other  categories  and  skill  sets. 


Question 

August  (VCCT) 
tTest  L  Pi'ob>|t| 

November  {VBS  2) 
tTest  L  Pi'ob>|t| 

1  CALL 

5.6930 

<0.0001 

1 .6593 

0.1075 

1  C  lED 

4.6351 

0.0001 

1 .4076 

0.1695 

1  PRO  1 

5.0091 

<0.0001 

1 .2223 

0.2311 

1  PRO  2 

4.2040 

0.0004 

1 .5323 

0.1359 

1  C  AMB 

4.8465 

<0.0001 

3.2621 

0.0028 

1  PROS 

4.4770 

0.0002 

2.5593 

0.0158 

1  PRO  4 

4.5917 

0.0001 

3.3211 

0.0024 

1  PROS 

4.5344 

0.0002 

0.3485 

0.7299 

1  C  WEA 

5.3531 

<0.0001 

2.6955 

0.0114 

1  PRO  6 

6.0694 

<0.0001 

1.0441 

0.3048 

1  PRO  7 

5.6000 

<0.0001 

2.8304 

0.0082 

1  PROS 

3.9056 

0.0008 

2.6833 

0.0117 

1  PRO  9 

4.8062 

<0.0001 

1 .5626 

0.1286 

1  PRO  10 

4.6986 

0.0001 

1 .0946 

0.2824 

1  CCOM 

5.503471 

<0.0001 

-1 .3374 

0.1912 

1  PRO  1 1 

5.3735 

<0.0001 

-1 .4384 

0.1607 

1  PRO  12 

4.9698 

<0.0001 

-0.6485 

0.5216 

1  PRO  13 

5.0070 

<0.0001 

-0.9411 

0.3542 

1  C  PRE 

5.0952 

<0.0001 

0.2314 

0.8186 

1  PRO  14 

3.2188 

0.0040 

-0.2543 

0.8010 

1  PRO  15 

3.7607 

0.0011 

-0.8915 

0.3798 

1  PRO  16 

3.8935 

0.0008 

1.2223 

0.2311 

1  PRO  17 

5.5235 

<0.0001 

0.7695 

0.4476 

Table  19.  Analysis  of  before  and  after  responses  to  statements  related  to  self- 
assessed  individual  proficiency  by  simulation  group;  yellow  highlight 
indicates  significant  increased  proficiency 
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In  summary,  we  observed  a  difference  between  changes  in  self- 
assessed  proficiency.  Regardless  of  skill  domain,  the  VCCT  group’s  self- 
assessed  proficiency  increased.  However,  the  VBS  2™  group’s  proficiency 
ratings  only  changed  for  two  skill  groups.  We  noted  no  decrease  in  self- 
assessed  proficiency  ratings  for  either  group  or  for  any  skill  category. 

c.  Unit  Tactical  Proficiency 

Just  as  participants  rated  individual  skill  sets  before  and  after  virtual 
training,  they  assessed  unit  proficiency  for  the  skill  sets  as  well.  We  used  the 
same  methodology  to  analyze  the  results,  starting  with  ANOVA  to  test  the  null 
hypothesis  that  changes  in  self-assessed  proficiency  were  the  same  by  treatment 
group.  As  shown  in  Table  20,  we  found  evidence  to  reject  the  null  hypothesis  in 
the  case  of  the  composite  item  and  all  subcategories  and  individual  skills  (F-ratio 
25.4943,  p-value  less  than  0.0001).  Using  a  t-test  to  investigate  the  null 
hypothesis  that  each  group  did  not  experience  a  change  in  self-assessed 
proficiency,  we  found  that  the  self-assessed  proficiency  ratings  for  all  unit 
composite  categories  and  individual  skills  increased  for  the  VCCT  group,  as 
shown  in  Table  21 .  On  the  other  hand,  self-assessed  proficiency  for  the  VBS  2^''^ 
group  decreased  overall.  Vehicle  recovery,  casualty  evacuation,  and 
communication  accounted  for  this  decrease;  self-assessed  proficiency  ratings 
had  no  change  for  the  other  skill  sets  for  the  VBS  2™  group. 
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# 

QUESTION 

F  RATIO 

P(>F) 

All 

COMPOSITE  UNIT  -  Tactical 

27.4943 

<0.0001 

I 

COMPOSITE  UNIT-IED 

23.2832 

<0.0001 

1 

UNIT  -  React  to  an  unexploded  Improvised  Explosive 
Device 

20.0585 

<0.0001 

2 

UNIT  -  React  to  an  Improvised  Explosive  Device 
detonation 

25.7603 

<0.0001 

1 

COMPOSITE  UNIT -Ambush 

15.8090 

0.0002 

3 

UNIT  -  Take  immediate  action  against  a  blocked 
ambush 

17.8025 

0.0001 

4 

UNIT  -  Take  immediate  action  against  an  unblocked 
ambush 

13.0047 

0.0007 

1 

5 

UNIT  -  Cordon  and  360  degree  security 

19.1706 

<0.0001 

1 

COMPOSITE  UNIT  -  Weapons 

22.9595 

<0.0001 

6 

UNIT  -  Employ  vehicle  machine  guns  /  weapons 

18.8279 

<0.0001 

7 

UNIT  -  Mounted  fire  and  maneuver 

21.2568 

<0.0001 

8 

UNIT  -  Shift  fires  /  cease  fires 

23.8085 

<0.0001 

1 

9 

UNIT  -  Vehicle  recovery  /  bump  plan 

27.5881 

<0.0001 

1 

10 

UNIT  -  Casualty  evacuation 

21.4048 

<0.0001 

1  1 

COMPOSITE  UNIT  -Communications 

32.0918 

<0.0001 

11 

UNIT  -  Communication  with  higher  headquarters 

31.1784 

<0.0001 

12 

UNIT  -  Communication  between  vehicles  in  a  convoy 

29.8752 

<0.0001 

13 

UNIT  -  Communication  between  personnel  in  vehicle 

30.2392 

<0.0001 

Table  20.  Analysis  of  before  and  after  responses  to  statements  related  to  self- 

assessed  unit  proficiency  by  individual  question;  yellow  highlight  indicates 
significant  difference  between  simulation  training  groups  with  VCCT  group 

having  the  higher  ratings 
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Question 

August  (VCCT) 
tTest  Prob>|t| 

November  {VBS  2) 
tTest  Prob>|t| 

U  C  ALL 

4.6770 

0.0002 

-2.3976 

0.0229 

U  C  lED 

4.8492 

0.0001 

-1.3713 

0.1804 

U  PRO  1 

4.4853 

0.0003 

-1.3128 

0.1992 

U  PRO  2 

5.1514 

<0.0001 

-1.3926 

0.1740 

U  C  AMB 

4.1309 

0.0006 

-0.9939 

0.3282 

U  PROS 

4.0105 

0.0008 

-1.3932 

0.1738 

U  PRO  4 

4.1576 

0.0006 

-0.5835 

0.5639 

U  PROS 

5.3785 

<0.0001 

-1.1791 

0.2476 

U  C  WEA 

5.2778 

<0.0001 

-1.8082 

0.0806 

U  PRO  6 

5.2050 

<0.0001 

-1.4850 

0.1480 

U  PRO  7 

5.0000 

<0.0001 

-1.6473 

0.1099 

U  PROS 

4.6599 

0.0002 

-2.1818  1 

0.0371 

U  PRO  9 

4.9242 

<0.0001 

-2.5668 

0.0155 

U  PRO10 

4.3507 

0.0003 

-2.1356 

0.0410 

U  CCOM 

4.3261 

0.0004 

-3.5634 

0.0012 

U  PR011 

4.0559 

0.0007 

-3.8545 

0.0006 

U  PR012 

4.3529 

0.0003 

-3.2—i 

liKso 

U  PR013 

4.3333 

0.0004 

-U« 

11.0027 

Table  21 .  Analysis  of  before  and  after  responses  to  statements  related  to  self- 
assessed  unit  proficiency  by  simulation  group;  yellow  highlight  indicates 
significant  increased  proficiency,  red  highlight  indicates  significant 

decreased  proficiency 


d.  Simulation  Group  Qualitative  Ratings 

As  in  Experiment  4,  the  participants  rated  statements  about  the 
simulation  training  experience  on  a  five-point  Likert  scale.  We  used  ANOVA  to 
investigate  the  null  hypothesis  that  the  two  simulation  groups  did  not  differ  in  their 
responses  to  these  statements.  For  five  of  the  statements,  the  VCCT  group’s 
responses  were  significantly  higher  than  those  of  the  VBS  2™  group.  Table  22 
highlights  these  responses. 
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# 

QUESTION 

F-RATIO 

P(>F) 

1 

This  training  mission  was  successful 

31.0227 

<0.0001 

2 

During  this  exercise,  1  felt  like  my  actions  in  the  virtual 
environment  had  no  consequences 

1.3791 

0.2458 

3 

During  this  exercise,  1  felt  like  1  was  playing  a  game 

0.7526 

0.3898 

4 

During  this  exercise,  1  felt  like  1  was  conducting 
training 

10.4479 

0.0022 

5 

During  this  exercise,  1  felt  like  1  was  part  of  a  group 
working  together 

10.9551 

0.0017 

6 

During  this  exercise,  1  felt  isolated  from  others 

2.5204 

0.1187 

7 

This  computer  simulation  provided  sufficient  audio 
cues  for  me  to  know  what  was  going  on 

4.0546 

0.0494 

8 

This  computer  simulation  provided  sufficient  visual 
cues  for  me  to  know  what  was  going  on 

1 .7890 

0.1871 

9 

The  training  value  of  this  exercise  came  from  the 
debriefing  and  not  the  exercise  itself 

20.3209 

<0.0001 

Table  22.  Analysis  of  before  and  after  responses  to  statements  related  to  the 
overall  simulation  training  experience;  yellow  highlight  indicates 
significant  difference  between  simulation  training  groups  with  VCCT 
group  having  the  higher  ratings 


e.  Comments 

Participants  provided  free  flowing  subjective  response  to  the 
training  through  open-ended  questions  on  the  last  page  of  the  survey.  These 
questions  asked  what  skills  were  easiest  and  most  difficult  for  the  individual, 
crew,  and  unit.  Additionally,  participants  responded  about  how  the  exercise 
helped  them  and  how  it  wasted  their  time.  Finally,  participants  had  the 
opportunity  to  provide  any  other  comments  they  felt  were  important. 

Given  the  statistically  proven  differences  in  attitude  toward 

simulation  and  self-assessed  proficiency  between  the  simulation  groups,  the 

comments  section  demonstrated  several  common  themes  between  the  two 

groups.  Communications,  including  crew  communications,  unit  communications, 

and  communications  to  higher  headquarters,  topped  the  list  of  concerns  for  both 

groups.  Marines  from  both  groups  expressed  concern  that  down  time  between 

scenarios  wasted  time.  Both  groups  found  situational  awareness  challenging, 
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while  they  considered  driving  and  shooting  very  easy  skills  in  the  simulation 
environment.  Marines  from  both  groups  felt  that  the  training  helped  to  provide  an 
overview  of  convoy  operations,  and  they  felt  that  more  of  the  training  was 
necessary  and  appropriate. 

Comments  differed  in  that  many  VBS  2™  participants  provided 
negative  overall  comments  such  as  “It  [VBS  2™  virtual  training]  did  not  help  the 
unit”  and  “patience  for  others  learning  controls”  was  a  difficult  individual  skill. 
Other  negative  comments  included  “too  little  time  to  learn  the  program,”  “no  one 
knew  what  was  going  on,”  and  “get  trained  administrators.”  The  theme  of  too 
little  training  time  was  very  common  throughout  the  VBS  2™  training  group  with 
twelve  participants  annotating  the  comment  somewhere  in  the  survey.  The  trend 
of  these  comments  did  not  criticize  the  simulation  itself,  but  rather  the 
administration  of  the  exercise  as  exemplified  by  the  following  comment:  “it  [VBS 
2™]  could  be  a  useful  tool”.  Participants  directed  some  of  the  criticism  toward 
themselves,  such  as  “student  discipline  was  poor.” 

3.  Instructor  Surveys 

The  instructor  cadre  filled  out  surveys  immediately  after  their  VBS  2™ 
training  and  then  again  after  conducting  the  VBS  2™  exercise  with  the  Marine 
class.  However,  only  five  instructors  filled  out  the  post-exercise  survey,  and 
because  of  administrative  problems  with  the  surveys,  we  could  only  match  two 
instructors’  pre-  and  post-exercise  surveys.  For  this  reason,  we  did  not  analyze 
most  of  the  data  from  the  surveys.  However,  it  is  worth  noting  that  instructors 
rated  their  own  proficiency  for  twelve  administrator  skills  immediately  after  their 
training.  All  eleven  instructors  rated  all  skills  with  a  three  or  higher  on  a  five-point 
Likert  scale. 

One  set  of  data,  shown  in  Table  23,  is  worthy  to  note,  even  though  it  only 
reflects  the  input  of  the  five  instructors  who  completed  the  post-exercise  survey. 
On  one  section  of  the  post-exercise  survey,  the  instructors  rated  whether  they 
preferred  VCCT  or  VBS  2™  for  each  of  the  tactical  skills  evaluated  in  the 
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experiment.  The  survey  used  a  five-point  Likert  scale  to  capture  strength  of 
preference  with  one  indicating  a  strong  preference  for  VBS  2™,  three  indicating 
no  preference  between  the  two,  five  indicating  a  strong  preference  for  VCCT,  and 
two  and  four  indicating  moderate  preferences.  We  averaged  the  ratings  for  each 
individual  skill  so  that  averages  below  three  indicated  an  overall  preference  for 
VBS  2™  and  averages  higher  than  three  indicated  a  preference  for  VCCT.  Most 
of  the  ratings  were  within  0.5  of  the  “no  preference”  mark  of  three.  However,  a 
pattern  emerged  in  the  grouping  of  the  averages.  For  tactical  action  skills  such 
as  weapon  employment  and  reaction  drills,  instructors  preferred  VCCT.  On  the 
other  hand,  instructors  preferred  VBS  2™  for  communications  and  tactical 
planning  skills.  Skills  that  stood  out  with  strong  preferences  included  a 
preference  for  VBS  2™  (average  2.0)  for  casualty  evacuation  and  a  preference 


for  VCCT  for  mounted  fire  and  maneuver  (4.0). 


# 

QUESTION 

AVERAGE 

ST  DEV 

1 

React  to  an  unexploded  Improved  Explosive 

Device 

3.5 

1.4 

2 

React  to  an  Improvised  Explosive  Device 
detonation 

3.5 

1.4 

3 

Take  immediate  action  against  a  blocked  ambush 

3.5 

1.4 

4 

Take  immediate  action  against  an  unblocked 
ambush 

3.5 

1.4 

5 

Cordon  and  360  degree  security 

2.4 

1.7 

6 

Employ  vehicle  machine  guns  /  weapons 

3.5 

1.5 

7 

Mounted  fire  and  maneuver 

4.0 

0.9 

8 

Shift  fires  /  cease  fires 

3.5 

1.5 

9 

Vehicle  recovery  /  bump  plan 

3.0 

1.7 

10 

Casualty  evacuation 

2.0 

1.1 

11 

Communication  with  higher  headquarters 

2.7 

1.6 

12 

Communication  between  vehicles  in  convoy 

2.7 

1.6 

13 

Communication  between  personnel  in  vehicle 

2.5 

1.5 

14 

Execute  the  troop  leading  steps  using  BAMCIS 

2.8 

1.6 

15 

Conduct  mission  analysis  using  METT-T 

2.8 

1.6 

16 

Receive  a  5  paragraph  operational  order 

2.8 

1.6 

17 

Give  a  5  paragraph  operational  order 

2.8 

1.6 

Table  23.  Analysis  of  instructor  preferences  for  VCCT  or  VBS  2™;  numbers  lower 
than  three  with  yellow  highlight  indicate  a  preference  for  VBS  2™  and 
numbers  higher  than  3.0  with  green  highlight  indicate  a  preference  for 

VCCT 
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Instructors  also  had  the  opportunity  to  provide  subjective  comments  at  the 
end  of  their  training  and  at  the  end  of  the  exercise.  Comments  at  the  end  of  the 
instructor  training  tended  to  be  positive  and  optimistic,  reflecting  a  confidence  in 
the  ability  to  run  an  exercise.  The  only  negative  trend  was  a  desire  for  more 
training  time  with  VBS  2™.  Comments  after  the  exercise  execution  reflected  a 
positive  attitude  about  the  training  potential  of  the  simulation.  However, 
instructors  expressed  concern  about  the  technical  aspects  of  the  training  with 
comments  such  as  “not  user  friendly,”  “too  many  moving  parts,”  and  “time 
consuming  setup.” 

4.  Notes  About  Experiment  Execution 

Some  notes  about  the  exercise  execution  are  helpful  in  order  to  properly 
understand  both  the  VCCT  and  VBS  2™  portions  of  the  experiment.  In  this 
experiment,  our  role  was  primarily  observational  leaving  little  opportunity  to 
change  the  course  of  the  exercise.  For  this  reason,  the  exercises  did  not  always 
progress  in  the  best  interests  of  the  study,  but  the  deviations  from  the  desired 
protocol  proved  instructive  in  themselves,  providing  insight  into  the  impact  of  the 
trainer  on  exercise  effectiveness. 

For  the  VCCT  exercise,  the  training  cadre  planned  two  runs  of  a  convoy 
scenario  with  Marines  changing  roles  for  the  second  run.  The  staff  had  made 
some  effort  to  organize  the  students  prior  to  the  event  with  a  designated  convoy 
commander  and  assistant  convoy  commander  ready  in  the  briefing  room  at  the 
beginning  of  the  exercise.  The  exercise  proceeded  with  no  difficulties  from  an 
administrative  point  of  view.  At  the  end  of  the  exercise,  the  training  cadre 
conducted  a  20-minute  after  action  review  session  using  the  VCCT  playback 
capability  administered  by  the  contractor  staff.  The  contractors  spent  a  half  hour 
re-setting  the  scenario  while  the  Marines  prepared  for  the  second  run.  However, 
the  contractors  could  not  get  the  second  run  of  the  scenario  to  work  properly,  so 
the  second  run  was  aborted  and  the  training  ended  about  an  hour  earlier  than 
anticipated. 
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Many  shortfalls  in  the  administration  of  the  training  hampered  the  VBS  2™ 
exercise.  The  instructor  cadre  had  set  up  the  computers  early  in  the  morning  of 
the  training  day.  The  computers  were  properly  networked  and  ready  from  a 
hardware  standpoint,  but  they  had  just  been  booted.  Only  one  instructor 
supported  the  training  exercise.  When  the  students  were  ready  to  train, 
communications  had  not  been  set  up  and  there  was  no  plan  to  do  so.  The 
scenario  was  not  loaded.  Without  even  a  convoy  commander  identified,  the 
student  class  was  not  organized  in  any  way  to  conduct  the  training.  The  staff 
had  planned  for  two  hours  of  VBS  2™  training,  which  left  a  very  short  time  to 
familiarize  Marines  with  the  interface,  introduce  a  scenario,  organize  the 
students,  conduct  the  training,  and  debrief  the  exercise.  Quite  simply,  the  time 
allotted  was  too  short.  Interface  training  was  insufficient.  Marines  received  no 
meaningful  brief,  the  scenario  was  rushed,  and  the  instructor  conducted  no  after 
action  review  afterward.  The  instructor  attempted  two  scenarios,  but  both  were 
convoluted  by  gross  situational  awareness  problems  that  were  exacerbated  by 
Marines  yelling  across  the  room  in  the  absence  of  a  communications  system. 
The  instructor  typically  focused  on  controlling  enemy  and  neutral  entities  in  the 
VBS  2™  environment,  neglecting  technical  difficulties,  higher  headquarters 
responsibilities,  and  evaluation  note  taking  and  critique.  In  summary,  the 
exercise  was  poorly  planned,  poorly  supported,  and  poorly  executed. 

D.  DISCUSSION 

1.  Hypothesis 

Taking  the  statistical  analysis  of  this  experiment  at  face  value,  we  find 

evidence  to  reject  the  null  hypothesis  of  the  experiment.  We  can  conclude  that  a 

trainer  who  has  been  trained  in  the  operation  of  the  simulation,  and  is  thereby 

able  to  control  the  simulation  exercise,  will  produce  training  that  is  less  effective 

to  that  produced  by  a  trainer  who  implements  the  simulation  exercise  through  a 

third  party.  Specifically,  a  trainer  who  implements  the  simulation  exercise 

through  a  third  party  will  produce  better  training  than  a  trainer  directly  using  the 
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simulation  tool.  The  improved  self-assessment  proficiency  ratings  of  the  VCCT 
group  support  this  conclusion.  Self-assessed  proficiency  increased  for  all  tactical 
skill  sets  for  the  VCCT  group  but  stayed  the  same  or  decreased  for  the  VBS  2™ 
group. 

However,  this  experiment  had  too  many  confounding  variables  to  justify 
such  a  conclusion  about  the  null  hypothesis.  Both  exercises  had  flaws  in  their 
execution  that  confounded  analysis.  The  poor  planning  and  execution 
associated  with  the  VBS  2™  exercise  made  the  data  collected  virtually  useless. 
Statistical  analysis  of  instructor  feedback  was  meaningless  because  of  the  small 
number  of  available  surveys.  In  short,  this  experiment  provided  no  means  to  say 
anything  definitive  about  the  null  hypothesis. 

However,  the  conduct  of  the  VBS  2™  exercise  provided  insight  into  the 
hypothesis.  The  crux  of  the  issue  examined  in  this  experiment  was  whether 
Marines  received  better  training  if  the  trainers  implemented  virtual  training 
themselves  or  through  a  third  party.  Clearly,  conducting  training  through  a  third 
party  with  VCCT  provided  a  better  training  experience  than  instructor 
administered  training  through  VBS  2™.  It  does  not  take  a  lot  of  statistical 
number  juggling  to  understand  why  we  saw  such  a  result  in  this  case.  The  VBS 
2™  training  was  poorly  planned  and  executed.  Most  likely.  Marines  would  prefer 
a  smoothly  executed  exercise  more  than  a  disorganized  one  no  matter  the  forum, 
skill  set,  or  venue.  Thus,  the  question  from  this  experiment  that  truly  bears 
discussion  is  why  the  VBS  2™  training  was  poorly  done. 

2.  Insights 

a.  Hammering  Without  Knowing  Where  the  Naii  is 

Virtual  training,  no  matter  what  computer  platform  is  used  or  who 
administers  it,  is  nothing  more  than  a  tool  to  accomplish  a  training  objective.  In 
this  regard,  the  virtual  training  platform  is  like  any  other  tool:  the  user  must  know 
how  it  works,  what  to  use  it  for,  how  to  fix  it  when  it  breaks,  what  support  it 
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needs,  etc.  Most  importantly,  the  user  must  know  what  he  or  she  intends  to 
accomplish  with  the  tool.  This  statement  seems  so  basic  that  it  threatens  to 
insult  the  reader’s  intelligence,  but  its  violation  caused  problems  in  the  execution 
of  the  VBS  2™  training  in  this  experiment. 

The  senior  enlisted  leader  of  EPME  Kaneohe  Bay  expressed  the 
training  goals  for  the  virtual  training  very  clearly.  He  sought  to  put  the  Marines  in 
a  tactical  situation  in  which  they  would  have  to  make  a  plan,  execute  the  plan, 
and  make  decisions  using  teamwork  and  leadership  in  order  to  accomplish  an 
objective.  Unlike  Experiment  4,  the  instructor  staff  did  not  specifically  seek  to 
improve  the  group’s  convoy  skills,  although  it  would  have  been  a  desirable  side 
effect.  Rather,  the  convoy  scenario  provided  a  tactical  environment  known  well 
enough  to  the  average  Marine  Sergeant  that  students  could  jump  in  and  start 
making  plans  and  decisions. 

The  execution  of  the  VBS  2™  exercise  reflected  a  loss  of  focus  on 
the  original  objective.  Instead,  the  instructors  became  focused  on  getting 
through  the  technical  details  of  administering  the  exercise.  Success  got  a  very 
different  definition,  with  instructors  simply  trying  to  get  through  all  of  the  wickets 
to  create  a  viable  training  environment  for  the  Marines.  Whether  or  not  the 
Marines  got  the  opportunity  to  make  reasonable  decisions  got  lost  in  the  turmoil 
of  making  the  event  happen.  Certainly,  the  focus  on  student  learning  was  lost 
when  instructors  completely  omitted  the  debrief,  where  decision  consequences 
would  logically  be  discussed  and  critiqued. 

On  the  other  hand,  the  VCCT  training  did  not  immediately 
demonstrate  symptoms  of  this  loss  of  focus  problem.  However,  it  is  also  difficult 
to  conclude  that  the  focus  was  clearly  there.  Rather,  the  instructors  used  a 
canned  training  program  that  the  contractor  staff  ran  routinely  for  many  different 
units.  The  contractors  knew  how  to  run  the  scenario,  had  seen  most  of  the 
variations  on  execution  Marines  would  make,  and  knew  how  to  run  the  playback 
tool  prompting  the  instructor  to  debrief.  The  appearance  of  accomplishing  the 
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objective  of  facilitating  a  decision-making  forum  was  more  the  result  of  contractor 
experience  than  instructor  cadre  intervention. 

In  summary,  a  trainer  who  forges  ahead  with  his  or  her  own  training 
program  must  do  so  with  clear  objectives  in  mind.  In  the  case  of  the  VBS  2™ 
training,  the  trainers  had  the  tool  and  knew  how  to  use  it.  However,  a  carpenter 
who  knows  how  to  use  a  hammer  but  just  beats  on  the  wood  accomplishes 
nothing.  He  must  know  where  the  nail  is  and  hit  it  with  each  stroke.  The  same  is 
true  of  trainer  administered  simulation  exercises. 

b.  Practice  Makes  Perfect 

When  we  trained  the  EPME  Kaneohe  Bay  instructor  staff  to  use 
VBS  2™,  we  spent  three  days  on  the  training  following  a  curriculum  similar  to  the 
VBS  2™  portion  of  DVTE  training  administered  by  the  contractors  at  the  I  MEF 
Simulation  Center  in  Camp  Pendleton.  Such  a  curriculum  is  adequate  to  teach 
the  basics  of  building  a  scenario,  using  the  real-time  editor  to  administer  an 
execution  run  of  the  scenario,  and  using  the  after  action  review  tool  to  facilitate 
debriefing.  However,  administering  a  VBS  2™  exercise  to  a  group  of  novice 
users  requires  quite  a  bit  of  knowledge.  The  trainer  must  be  able  to  apply  that 
knowledge  on  his  or  her  feet.  There  is  no  time  to  consult  other  people  or 
documentation  for  an  answer,  and  there  is  no  liberty  to  experiment  and  figure  out 
the  answer.  Once  the  exercise  starts  running,  the  administrator  must  know  what 
he  or  she  is  doing  and  be  on  his  or  her  toes  at  all  times  to  keep  the  environment 
realistic  for  the  training  audience. 

The  three  days  of  VBS  2™  training  is  clearly  insufficient  to  provide 
this  level  of  application  knowledge  for  the  average  Marine  trainer.  It  is  not  that 
more  training  is  required,  but  rather  that  the  trainer  must  have  the  opportunity  to 
practice  on  a  live  training  audience.  All  of  the  trainers  in  this  experiment 
improved  surprisingly  quickly  and  came  out  of  the  experience  ready  to  do  a  much 
better  job  the  next  time.  Not  only  would  such  experience  improve  the  next 
execution,  but  it  would  support  much  better  planning  as  well.  The  trained  but 
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unpracticed  VBS  2™  trainer  is  really  not  trained  at  all.  Only  practical  experience 
can  give  the  trainer  the  wherewithal  to  facilitate  good  training  with  a  tool  such  as 
VBS  2™. 

Nowhere  did  the  issue  of  practice  become  more  apparent  than  in 
time  allocation.  The  two-hour  period  allotted  for  the  VBS  2™  convoy  exercise 
was  far  too  short,  even  for  seasoned  administrators  and  users.  The  instructor 
cadre  had  not  accounted  for  the  time  to  load  scenarios  and  set  up  the  networks. 
They  had  no  experience  with  their  scenarios  to  gauge  how  long  an  exercise  run 
might  take.  The  resulting  plan  was  a  recipe  for  trouble  from  the  beginning 
because  every  step  of  the  process  had  to  be  rushed.  With  practice,  instructors 
would  not  have  attempted  such  a  plan  and  would  have  made  adjustments  to 
provide  a  better  chance  for  success. 

c.  Spreading  the  Trainer  Too  Thin 

Three  distinct  roles  are  necessary  to  implement  virtual  training 
through  a  simulation  platform  like  VBS  2™.  First,  someone  must  be  the 
facilitator.  In  this  role,  the  trainer  has  tasks  such  as  making  sure  everyone  gets 
networked  into  the  scenario,  causing  civilian  and  enemy  entities  to  act  in 
accordance  with  the  training  plan,  and  remembering  to  record  the  scenario  and 
bookmark  it  for  easy  review.  Second,  someone  must  be  the  technician.  Almost 
certainly,  some  members  of  the  training  audience  will  experience  technical 
difficulties.  The  problems  may  occur  because  of  computer  platform  issues, 
networking  problems,  or  software  glitches.  The  training  audience  may  induce 
problems  through  lack  of  experience,  failed  attempts  to  correct  a  minor  problem, 
or  accidental  actions  with  unintended  consequences.  Whatever  the  source,  the 
technician  must  handle  technical  problems  so  that  “computer-isms”  do  not 
overshadow  the  training.  Third,  someone  must  be  the  planner/evaluator.  The 
planner/evaluator  determines  the  training  objectives  and  designs  or  chooses  the 
scenario  to  meet  them.  In  the  planner/evaluator  role,  the  trainer  acts  as  one  or 
multiple  higher  headquarters  personnel.  The  planner/evaluator  must  take  notes 
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about  the  exercise  execution.  Most  importantly,  the  planner/evaluator  must 
observe  both  the  exercise  itself  as  it  unfolds  in  the  virtual  environment  and  the 
training  audience  as  they  work  through  the  execution.  This  observation  includes 
close  scrutiny  of  the  primary  communications  nets.  For  the  scope  of  exercises 
covered  in  this  thesis,  a  single  trainer  can  fulfill  all  three  of  these  roles  capably  as 
long  as  he  or  she  maintains  adequate  control  over  the  training  audience.  More 
likely,  a  training  staff  of  two  or  more  Marines  should  administer  the  exercise,  and 
the  roles  can  be  divided  between  available  personnel.  Whether  different  people 
split  the  roles  or  a  single  individual  performs  them  alone,  all  three  roles  must  be 
properly  handled  for  an  exercise  to  produce  successful  training. 

In  the  case  of  the  VCCT  training  administered  through  a  third  party, 
the  trainer  is  absolved  of  facilitator  and  technician  duties.  While  the  contractor 
handles  these  two  roles,  the  trainer  is  free  to  focus  on  planner/evaluator  tasks. 
The  cost  to  the  trainer  is  that  direct  facilitation  is  no  longer  possible  and 
adjustments  must  be  described  to  and  interpreted  by  a  third  party. 

The  division  of  labor  problem  became  painfully  obvious  in  the 
execution  of  the  VBS  2™  training.  The  trainer  focused  on  the  facilitator  role  to 
the  exclusion  of  both  the  technician  and  planner/evaluator  roles.  We  can  explain 
this  very  simply.  The  facilitator  role  is  fun.  Never  does  an  individual  have  such 
power  as  when  facilitating  a  virtual  exercise.  The  facilitator  is  right  in  the  thick  of 
the  action  making  things  happen  every  moment  of  the  exercise.  Once  one  learns 
the  interface,  it  is  easy  to  create  complex  problems  for  the  training  audience,  and 
the  effect  is  immediate  and  satisfying.  Such  a  role  appeals  to  the  aggressive, 
controlling  mindset  of  the  typical  Marine  leader.  Keeping  notes  about  vehicle 
dispersion  or  figuring  out  why  someone’s  mouse  is  not  working  simply  fall  to  the 
wayside  when  artillery  is  a  finger  click  away  and  the  enemy  RPG  team  can  be 
hidden  just  a  little  bit  better.  Nevertheless,  no  matter  the  appeal  of  any  particular 
role,  all  three  roles  must  receive  equal  emphasis  or  chaos  will  eventually  begin  to 
take  over. 
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Thus,  the  number  of  trainers  who  implement  VBS  2™  training  is 
important.  Curiously,  in  the  VCCT  exercise  where  contractors  capably  filled  the 
facilitator  and  technician  roles,  the  EPME  instructor  staff  maintained  at  least  two, 
and  usually  three,  instructors  to  implement  the  training.  On  the  other  hand,  when 
all  of  the  demands  of  virtual  training  implementation  rested  on  the  instructor  staff 
alone  for  the  VBS  2™  training,  the  instructor  staff  allocated  only  one  trainer  to  the 
exercise.  For  a  novice  staff,  this  personnel  level  is  insufficient.  At  a  minimum, 
one  person  needs  to  be  tied  to  the  control  station  computer  while  someone  else 
is  available  to  move  about  the  training  floor  to  help  the  training  audience. 

3.  Summary 

We  do  not  believe  that  Experiment  5,  when  viewed  in  total,  supports  any 
determination  about  the  null  hypothesis.  Unfortunately,  the  execution  of  the  VBS 
2™  exercise  was  too  flawed  to  properly  investigate  the  effects  of  direct  trainer 
facilitation  of  a  virtual  training  exercise.  However,  the  flaws  in  the  execution 
themselves  illustrated  the  challenges  of  the  trainer  staff  simultaneously  filling  the 
facilitator,  technician,  and  planner/evaluator  roles  of  a  virtual  training  exercise. 
Experiment  5  demonstrated  the  importance  of  objective-oriented  training, 
practice,  and  time  and  support  personnel  allocation  on  virtual  training  directly 
administered  by  the  trainer.  Future  work  could  shed  light  on  the  potential  of 
training  in  which  these  challenges  are  overcome.  However,  training  administered 
through  a  third  party  circumvents  these  problems  to  a  great  extent,  thereby 
reducing  the  risk  of  a  poorly  executed  training  event. 
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VI.  RECOMMENDATIONS  AND  CONCLUSIONS 


A.  CONCLUSIONS 

Based  on  the  results  of  Experiment  4,  we  conclude  that  serious  games 
such  as  VBS  2™  are  valid  training  tools  for  small  unit  tactics.  Serious  games  do 
not  necessarily  produce  better  results  than  traditional  training  methods,  but  they 
perform  at  least  as  well.  Serious  games  show  an  edge  over  traditional  training 
methods  in  user  satisfaction,  possibly  because  they  satisfy  the  technical  whims 
of  today’s  youth.  From  our  observations,  serious  games  are  not  a  single  answer 
to  training  needs,  but  rather  a  training  multiplier  to  use  with  other  training 
methods  such  as  the  sand  table  or  tactical  decision  game.  Serious  games  are  a 
viable  tool  for  the  trainer’s  toolbox  and  are  well  worth  the  time,  money,  and  effort 
to  develop  as  core  assets  to  a  unit’s  or  school’s  tactical  training  curriculum. 

Serious  games  come  with  a  price,  however.  The  trainer  must  understand 
how  to  use  the  serious  games  as  an  extension  of  his  or  her  training  skills. 
Experiment  5  provided  insight  into  a  representative  situation  that  demonstrated 
that  a  trainer’s  use  of  serious  games  is  a  learned  and  practiced  skill  rather  than  a 
natural  and  intuitive  technique.  The  trainer  must  be  able  to  clearly  articulate 
training  objectives.  Then,  the  trainer  must  understand  the  capabilities  and 
limitations  of  the  serious  game  well  enough  to  translate  the  training  objectives 
into  an  implementable  serious  game  exercise.  For  the  serious  game  training  to 
be  successful,  some  combination  of  people  must  fill  the  roles  of  facilitator, 
technician,  and  planner/evaluator.  The  trainer  must  fill  the  last  role  and  is  better 
served  knowing  as  much  as  possible  about  the  first  two.  We  did  not  find 
evidence  to  support  the  notion  that  the  trainer’s  involvement  in  the  facilitator  role 
can  improve  the  training  experience,  but  we  think  this  idea  deserves  further 
investigation.  However,  we  did  find  evidence  to  support  the  conclusion  that  the 
trainer  cannot  focus  on  the  facilitator  role  to  the  exclusion  of  the  other  roles;  such 
a  practice  leads  to  a  chaotic  evolution. 
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We  conclude,  then,  that  the  technical  aspects  of  serious  games  are  no 
longer  the  primary  issue  in  providing  innovative  and  effective  small  unit  tactical 
training  for  Marines  and  soldiers.  The  driving  issue  at  hand  is  developing  the  skill 
and  experience  of  the  trainer  to  use  serious  games  effectively.  We  conclude  that 
the  training  establishment  must  think  out  such  a  program  carefully,  plan  it, 
support  it,  monitor  it,  and  supervise  it.  A  program  that  simply  makes  the  training 
tool  available  to  the  trainer  without  actively  changing  the  way  a  trainer  develops 
his  or  her  training  program  risks  wasting  the  potential  of  the  tool,  wasting  the  time 
of  the  training  audience,  and  wasting  the  valuable  support  dollars  that  brought 
serious  games  into  the  limelight. 

A  training  transfer  study  seeks  to  determine  whether  a  given  training 
device  makes  an  individual  more  effective  or  proficient  in  a  certain  task  or 
mission.  Tradition  implies  that  this  analysis  should  focus  on  the  trainee. 
Throughout  the  course  of  this  thesis  adventure,  we  did  just  that  with  VBS 
that  is,  we  demonstrated  an  improvement  in  self-assessed  proficiency  within  a 
given  skill  set.  More  broadly,  we  determined  that  serious  games  are  as  effective, 
but  not  necessarily  more  so,  than  other  training  methods.  If  serious  games  are 
as  good  but  no  better  than  anything  else,  then  the  question  boils  down  to  what  is 
better  for  the  trainer.  We  conclude  that  it  is  time  for  the  era  of  the  trainee 
focused  training  transfer  study  to  close  for  serious  games.  The  focus  of  training 
transfer  study  in  the  serious  game  domain  needs  to  turn  to  the  trainer.  Such  a 
study  should  determine  whether  the  serious  game  makes  the  trainer  more 
effective  in  his  or  her  mission.  If  the  trainer  cannot  achieve  training  objectives 
more  efficiently,  serious  games  do  not  offer  enough  game  to  make  them  worth 
the  time,  money  and  hype.  We  firmly  believe  that  serious  games  offer  the  trainer 
a  whole  new  world  of  capability,  but  the  trainer  must  be  well  educated  in  the 
employment  of  the  tool  to  realize  this  potential. 
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B.  OBSERVATIONS 


This  thesis  provided  a  unique  opportunity  for  comment  on  serious  games. 
In  this  project  a  Marine  Corps  field  grade  officer  learned  to  administer  training 
with  VBS  2™  through  experience  and  self-instruction.  All  of  our  experiments 
were  done  with  what  we  could  learn  from  VBS  2™  as  it  came  out  of  the  DVTE 
boxes.  Resources  only  included  the  manuals  on  the  computers,  the  VBS  2™ 
website,  and  the  on-line  VBS  2™  support  forum.  We  got  a  three-day  summary 
view  of  the  I  MEF  Simulation  Center  contractor-led  DVTE  training  for  comparison 
purposes  after  we  had  completed  the  bulk  of  this  project.  No  outside  training 
influenced  our  work.  For  this  reason,  the  comments  of  an  experienced  trainer 
with  the  educational  background  provided  by  the  NFS  MOVES  program  who  has 
implemented  training  using  DVTE  “out  of  the  box”  should  have  value  to  the 
tactical  training  community  as  well  as  simulation  professionals. 

1.  Making  VBS  2™  an  Ubiquitous  Trainer 

At  one  time,  in  the  early  1990s,  Marine  Corps  officers  approached 
commercial  presentation  software,  such  as  Microsoft  PowerPoint,  with  caution 
and  trepidation.  Officers  did  not  know  how  to  use  the  software.  They  were  not 
used  to  seeing  information  presented  with  it.  Thus,  they  looked  at  the  software 
with  reluctance  and  passed  any  duties  associated  with  it  to  the  lowest  capable 
individual  they  could  find.  Now,  most  officers  use  such  software  quite  easily,  and 
modern  audiences  expect  presentations  developed  with  such  software  as  a 
means  of  information  exchange  and  consider  these  products  the  norm.  This 
change  occurred  for  two  main  reasons.  First,  Marines  have  learned  how  to  use 
the  software,  at  least  at  a  basic  level,  through  experience  and  observation. 
Second,  Marines  generally  do  not  have  to  build  presentations  from  scratch. 
Often,  presenters  provide  information  using  presentation  software  formats  that 
have  been  developed  and  refined  over  the  years,  and  Marines  simply  have  to 
cut,  paste,  and  update  to  achieve  their  objectives.  In  this  way,  neither  the 
learning  curve  nor  the  development  curve  is  steep,  so  the  software  is  much  more 
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palatable.  The  result  is  that  in  today's  world,  many  would  not  know  how  to 
accomplish  their  jobs  if  the  software  suddenly  disappeared. 

VBS  2™  is  at  the  same  point  that  presentation  software  was  two  decades 
ago.  Both  the  learning  curve  and  the  development  curve  are  steep.  Moreover, 
VBS  2™  is  not  the  method  with  which  people  are  used  to  having  their  training 
presented.  The  idea  that  personal  computer-based  training  will  one  day  reach 
the  level  of  ubiquitousness  enjoyed  by  presentation  software  is  not  far-fetched, 
however.  The  question,  then,  is  how  to  shorten  the  learning  and  development 
curves. 

A  good  way  to  shorten  the  learning  curve  is  through  schools.  The  learning 
curve  stays  long  if  no  one  understands  what  the  product  is  and  if  nothing  is  done 
to  teach  it.  The  DVTE  program  provides  a  training  package  for  unit 
representatives  upon  dissemination  of  the  gear,  but  this  program  does  not  lend 
itself  to  ubiquitousness.  It  only  temporarily  provides  someone  who  can  turn  on 
the  machines.  Without  a  program  in  place  to  continue  training,  this  temporary  fix 
will  not  live.  Ubiquitousness  occurs  through  an  organized  approach  that 
systematically  touches  everyone  in  the  Marine  Corps  at  one  point  or  another. 
Resident  schools  provide  such  an  opportunity.  Resident  schools  have  a 
standardized  hierarchy  organized  through  TECOM.  Students  in  resident  schools 
have  the  time  and  motivation  to  focus,  so  instruction  does  not  get  lost  in  a  series 
of  competing  unit  demands.  Finally,  schools  can  provide  students  with  the 
chance  to  experiment  and  ask  questions.  Not  all  Marines  go  to  the  appropriate 
resident  school  for  their  grade.  However,  if  VBS  2™  and  other  simulations  were 
taught  in  the  schools  for  NCOs,  SNCOs,  company  grade  officers,  and  field  grade 
officers,  units  would  have  a  sprinkling  of  simulation  interest  to  get  personal 
computer-based  training  out  of  its  infancy.  However,  the  Marine  Corps  and  other 
DoD  services  must  approach  this  training  in  schools  with  discretion.  Junior 
leaders  do  not  need  to  focus  on  the  key  that  makes  the  avatar  move  forward. 
Rather,  they  need  to  focus  on  how  to  use  the  simulation  as  an  extension  of  their 
training  capabilities.  This  includes  the  technical  ability  to  develop,  run,  and 

180 


debrief  scenarios  as  well  as  the  ability  to  integrate  the  simulation  into  the  unit’s 
training  schedule  to  provide  a  complementary  medium  to  enhance  the 
achievement  of  unit  objectives. 

The  armor  community  offers  an  example  of  this  concept  in  practice.  For 
both  the  Marine  Corps  and  Army  M1A1  tank  communities,  a  Master  Gunner 
serves  as  the  expert  for  all  matters  related  to  tank  gunnery.  This  enlisted  tanker 
goes  to  an  intense  school  at  Fort  Knox,  Kentucky  to  learn  about  all  aspects  of 
making  the  tank  put  main  gun  and  machinegun  rounds  on  target.  Part  of  this 
training  includes  instruction  about  the  use  of  the  Advanced  Gunnery  Training 
System  (AGTS),  the  simulator  to  train  the  gunner  and  tank  commander  in  crew 
level  gunnery  skills.  Not  only  does  the  Master  Gunner  learn  to  use  AGTS,  but  he 
learns  to  develop  a  company  level  gunnery  training  plan  that  includes  AGTS  as  a 
core  component  of  the  annual  training  cycle.  There  is  no  reason  that  training 
with  serious  games  cannot  follow  such  a  model,  and  we  recommend  scrutinizing 
this  concept  for  potential  ideas  about  incorporating  VBS  2™  into  training 
curricula. 

The  training  establishment  can  shorten  the  development  curve  through  an 
organized  proliferation  of  user  generated  simulation  products.  Users  should  not 
be  creating  their  training  from  scratch.  If  one  unit  puts  together  a  solid  simulation 
training  package  for  a  certain  training  objective,  there  is  no  reason  someone  else 
should  do  the  same  work  over  again.  However,  the  creation  of  an  environment  in 
which  users  can  pool  their  work  spawns  a  variety  of  obstacles  that  must  be 
overcome.  First,  one  must  be  able  to  identify  what  the  scenario  trains.  The 
mechanism  for  achieving  this  is  not  complex  or  new.  The  system  of  MPSs  in  the 
Marine  Corps  provides  an  adequate  mechanism  for  tagging  the  training 
objectives  of  a  given  scenario.  Second,  one  must  be  able  to  identify  what  is  in 
the  scenario.  This  information  is  not  difficult  to  organize  since  it  includes 
standard  data  that  everyone  understands  such  as  number  of  players,  number 
and  type  of  vehicles  and  weapons,  type  of  terrain,  and  type  of  enemy  situation. 
Third,  one  must  be  able  to  provide  quality  control  for  the  scenarios.  A  novice's 
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best  work  may  be  a  useless  product  for  the  experienced  trainer.  Some  authority 
must  separate  the  wheat  from  the  chaff  without  creating  an  approval  bureaucracy 
so  thick  that  everyone  is  scared  away.  Finally,  one  must  provide  quick  and  easy 
accessibility.  The  information  age  has  solved  any  technical  problem  in  this 
regard,  but  the  internal  administrative  issues  of  military  information  technology 
are  not  designed  with  quick  and  easy  accessibility  in  mind.  Through  DVTE, 
Marine  Corps  VBS  2™  lives  in  a  computer  environment  entirely  cordoned  off 
from  the  Internet  environment,  and  the  Navy  Marine  Corps  Intranet  (NMCI)  does 
not  offer  easy  solutions  to  this  challenge. 

With  formalization  of  standard  training  packages,  the  Marine  Corps 
training  community  can  implement  learning  management  software  to  help 
trainers  monitor  the  unit’s  progress.  We  believe  that  rigid  emphasis  on  such 
tools  can  stifle  training  initiative  and  create  programs  that  merely  put  a  check  in 
the  box.  Rather,  a  learning  management  tool  should  provide  trainers  with  a 
picture  of  current  and  desired  training  levels.  Then  the  tool  should  recommend 
ways  to  make  up  the  difference.  Serious  games  such  as  VBS  2™  lend 
themselves  to  such  management  programs,  and  these  programs  would  help 
institutionalize  serious  games  as  credible  training  platforms. 

The  Joint  Training  Counter-Improvised  Explosive  Device  Operations 
Integration  Center  (JTCOIC)  uses  VBS  2™  extensively.  The  organization  has 
collected,  organized,  and  documented  its  VBS  2™  models,  maps,  scripts,  and 
scenarios  and  makes  them  available  through  regular  e-mail  dissemination. 
JTCOIC  has  no  charter  to  act  as  a  DoD  VBS  2™  clearinghouse,  but  their  efforts 
at  organizing  VBS  2™  material  provide  a  model  for  the  rest  of  the  military  VBS 
2™  community.  Expanding  on  this  model  DoD  wide  could  benefit  all  users 
exponentially. 

2.  Making  a  Serious  Game  Trainer 

In  our  Experiment  5  comments,  we  pointed  out  that  a  trainer  using  VBS 
2™  must  be  a  facilitator,  technician,  and  planner/evaluator  or  have  such 
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personnel  on  his  or  her  staff.  Commanders  with  DVTE  in  their  inventories  must 
consider  personnel  when  determining  how  to  implement  the  simulations  at  their 
disposal.  Certainly,  junior  enlisted  personnel  should  be  trained  to  fill  the 
technician  role  of  serious  game  training.  Equally  as  clear,  senior  SNCOs  and 
junior  officers  should  fill  the  planner/evaluator  role.  However,  commanders 
cannot  depend  on  junior  enlisted  personnel  to  fill  the  facilitator  role  and  must 
ensure  that  SNCOs  and  junior  officers  are  suited  to  either  fill  this  role  or  closely 
supervise  it.  We  believe  the  power  of  virtual  training  lies  in  the  judicial 
application  of  the  facilitator  role  and  the  transfer  of  this  role  to  the  trainer. 
Commanders  can  guarantee  this  by  getting  trainers  to  DVTE  training  as  well  as 
junior  enlisted  personnel  who  will  serve  as  “pucksters.” 

3.  Appreciation  of  VBS  2™  in  the  Fieet 

The  generation  of  officers  and  upper  level  federal  employees  who  are 
buying  today's  training  systems  are  technically  adept  and  progressive  enough  to 
understand  that  today's  youth  expect  a  technically  based  training  experience 
from  the  technology  revolution.  However,  they  do  not  necessarily  understand 
this  wired  generation.  The  exuberance  to  appease  the  young  serviceman  who 
has  never  lived  without  Internet  or  computers  has  led  to  a  variety  of  assumptions 
that  may  not  be  true  and  may  have  never  been  true.  Because  of  the  pace  of 
technological  change,  the  truth  behind  these  assumptions  may  change  too  fast  to 
ascertain  their  validity  at  any  point  in  time. 

For  example,  one  might  surmise  that  today's  generation  would  appreciate 
personal  computer-based  gaming  for  training.  This  assumption  seems  to  fit  with 
the  stereotype  of  the  data-device-wielding  teenager.  Furthermore,  one  can  find 
numerous  data  to  support  the  idea  that  today's  youth  play  video  games. 
However,  data  device  use  does  not  connote  a  gamer,  and  a  gamer  does  not 
connote  appreciation  of  serious  games  as  training  devices.  A  similar  domain 
illustrates  this  concept.  Almost  everyone  has  seen  a  movie,  most  people  have 
seen  many  movies,  and  many  people  enjoy  movies  as  a  form  of  entertainment. 
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However,  that  does  not  mean  that  Marines  enjoy  watching  military  films  for 
training.  Even  if  Marines  like  movies  with  military  themes,  they  most  likely  enjoy 
qualities  of  the  movie  that  lend  to  entertainment  rather  than  realism.  We  must 
remember  this  concept  when  we  consider  serious  games  and  today's 
technophilic  youth. 

Even  for  the  avid  gamer,  we  cannot  assume  an  immediate  appreciation  of 
serious  games  for  training.  Often,  the  gamer  is  the  one  who  dislikes  the  training 
simulation  the  most.  Most  people  who  do  not  bicycle  would  consider  any  bicycle, 
regardless  of  style,  source,  quality,  age,  or  other  factors,  the  same.  On  the  other 
hand,  an  avid  cyclist  would  certainly  never  use  a  mountain  bike  for  road  racing 
and  would  quite  likely  seek  the  highest  end  racing  bike  money  could  buy  if  road 
racing  was  the  objective.  Gaming  is  no  different.  Different  games  have  different 
purposes,  qualities,  and  styles.  Gamers  often  bring  a  set  of  expectations  to  the 
training  simulation  that  the  military  cannot  satisfy  in  such  a  dynamic  industry. 
The  barracks  XBox  jockey  might  be  the  very  person  most  put  off  by  the  personal 
computer-based  simulator  chosen  by  the  military  service. 

In  our  enthusiastic  theorizing  about  serious  games,  we  might  be  tempted 
to  imagine  a  day  when  military  personnel  train  in  virtual  environments  of  their 
own  volition  and  initiative  because  they  enjoy  the  game.  Our  experience  in  this 
study  showed  that,  left  alone.  Marines  like  to  do  in  a  simulation  environment  what 
they  like  to  do  anywhere  else  when  left  alone  to  their  own  devices.  They  like  to 
fool  around.  They  like  to  work  as  independent  players,  getting  the  best  of  their 
mates.  They  like  to  have  fun.  While  some  of  this  play  may  lead  to  tactical 
lessons  learned,  unit  training  can  only  be  realized  through  a  disciplined 
application  of  training  developed  according  to  an  organized  set  of  objectives. 
Simulation  training  cannot  eliminate  the  need  for  the  trainer  any  more  than  it  can 
eliminate  the  need  for  live  training. 

We  do  not  record  these  observations  to  suggest  that  personal  computer- 

based  training  is  a  lost  cause.  Quite  the  contrary,  today's  military  is  not  surprised 

that  it  exists  and  readily  embraces  it.  Arguably,  they  expect  it  and  have  high 
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standards  for  it.  However,  we  cannot  expect  them  to  appreciate  the  training 
simply  because  we  were  progressive  enough  to  militarize  a  computer  game. 
Today's  youth  expects  both  technical  quality  and  training  quality  from  the 
simulation,  as  they  should.  This  demands  an  adept  trainer  who  knows  how  to 
use  the  simulation  as  an  extension  of  his  training  abilities.  It  also  demands  a 
simulation  architecture  that  can  keep  up  with  the  times,  modernizing  training 
tools  at  the  rate  that  Marines  update  the  equipment  in  their  barracks  rooms  and 
living  rooms  from  the  local  gaming  superstore. 

4.  Counting  Dollars 

In  Experiment  4,  we  demonstrated  that  VBS  2™  can  train  Marines  as 
effectively  as  traditional  means  such  as  the  sand  table  for  small  unit  tactical 
skills.  This  work  shows  why  training  with  serious  games  should  be  considered  in 
a  resource  constrained  environment  that  precludes  live  training.  However,  the 
findings  suggest  that  the  trainer  can  consider  traditional  training  on  an  equal 
footing  with  serious  game  training.  Certainly,  when  it  comes  down  to  dollars  and 
cents,  the  operational  trainer  must  choose  to  buy  a  suite  of  computers  with  a 
vision  toward  distinct  and  unique  training  gain.  We  believe  serious  game  training 
offers  advantages  over  traditional  training  garrison  exercises.  While  we  do  not 
propose  to  use  this  thesis  as  a  cost  analysis  vehicle  for  serious  games,  we 
suggest  the  following  advantages  of  serious  game  technology. 

a.  Users  Like  It 

Experiment  4  demonstrated  that  Marines  preferred  using  VBS  2™. 
Subjectively,  Marines  felt  their  tactical  skills  improved  more  using  VBS  2™  than 
using  traditional  training  methods.  Despite  unique  gamer  biases,  training  aligned 
with  the  modern  technical  age  appeals  to  young  warfighters.  Marines  and 
soldiers  are  used  to  computers  in  their  daily  life,  and  they  accept  training 
administered  through  them.  As  Experiment  5  demonstrated,  this  acceptance 
cannot  be  taken  for  granted;  Marines  still  expect  quality  training  in  virtual 
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environments  like  anywhere  else.  Marines  like  to  train  live,  but  when  this  is  not 
possible,  they  like  using  well  administered  serious  games 

b.  The  After  Action  Review  Tooi  Expands  the  Trainer’s 
Options 

In  live  training,  the  exercise  proceeds,  and  then  it  is  gone.  Only 
memories  and  notes  remain  after  its  completion.  Traditional  training  like  the 
sand  table  produces  the  same  results.  However,  serious  games  such  as  VBS 
2™  offer  the  ability  to  record  the  exercise  and  view  it  exactly  as  it  happened.  As 
a  matter  of  fact,  the  trainer  can  review  the  exercise  from  many  different 
perspectives,  focusing  on  points  of  interest  in  whatever  manner  he  chooses. 
This  review  capability  is  unique  to  virtual  environments,  and  it  offers  the  trainer 
capabilities  that  cannot  be  achieved  through  any  other  means. 

c.  Events  Are  More  Reai 

Virtual  training  offers  the  potential  for  partial,  or  possibly  complete, 
immersion  in  the  training  environment.  Trainers  cannot  achieve  anything  but  the 
most  remote  level  of  immersion  using  sand  tables  or  tactical  decision  games. 
While  virtual  training  environments  cannot  achieve  enough  immersion  to  rival  a 
live  exercise,  they  provide  the  only  opportunity  for  immersion  out  of  the  choices 
for  garrison  training.  Trainees  can  experience  weapons  effects,  death,  the  visual 
effects  of  motion,  the  sounds  of  combat,  and  other  battlefield  effects.  While 
these  virtual  experiences  are  certainly  far  from  real,  they  create  a  more  realistic 
evolution  than  traditional  means.  Arguably,  some  effects,  such  as  death  and 
injury,  are  even  more  real  than  live  training.  This  unique  version  of  realism 
demands  that  serious  games  be  included  in  tactical  training  curricula. 

d.  Training  Can  Be  Standardized 

Computers  offer  dependability  that  training  based  on  human 
interaction  can  never  achieve.  Once  a  trainer  develops  a  valid  training  regimen 
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in  a  serious  game,  the  trainer  can  administer  the  scenario  the  same  way  every 
time.  The  trainer  can  export  the  scenario  to  other  trainers  easily.  This  unique 
advantage  of  computer-based  training  provides  the  opportunity  to  align  training 
scenarios  with  doctrine  and  then  mass  produce  them  for  the  convenience  of 
many  training  audiences.  Neither  live  nor  traditional  training  can  achieve  this 
level  of  standardization,  and  trainers  can  serve  their  communities  well  by 
capitalizing  on  this  feature. 

e.  Serious  Games  Offer  Ties  to  Higher  Headquarters 

Higher-level  command  cells,  particularly  battalion  and  above,  tend 
to  rely  more  and  more  on  simulation  to  achieve  their  training  goals.  Mobilizing 
large  units  for  training  is  costly  and  manpower  intensive.  Simulation  offers  the 
opportunity  for  large  staffs  to  exercise  procedures  and  techniques  while  avoiding 
these  costs.  Serious  games  allow  the  small  unit  tacticians  to  interface  with  these 
higher  level  simulations.  Such  interoperability  sets  the  stage  for  a  symbiotic 
relationship.  Small  units  have  the  benefit  of  operating  in  a  domain  supervised  by 
a  higher  headquarters  with  all  of  its  help  and  hindrances  while  higher  level  units 
have  the  added  chaos  of  subordinate  units  making  decisions  at  their  level  with  a 
thinking  mind  instead  of  a  computer’s  artificial  intelligence.  When  the 
Lieutenants  and  the  Lieutenant  Colonel  need  to  train  together,  simulation  offers 
an  outstanding  medium  to  accomplish  goals  without  wasting  the  valuable  time  of 
a  whole  unit  full  of  Marines  or  soldiers. 

5.  Shopping  for  Serious  Games 

An  interest  in  the  potential  of  transfer  studies  as  decision  drivers  in  the 
acquisition  of  training  games  partially  motivated  this  thesis  effort.  When 
choosing  a  serious  game  for  training  purposes,  the  military  trainer  can  use  four 
methods  to  select  the  appropriate  platform.  First,  the  trainer  can  conduct  a 
training  transfer  study  like  the  project  documented  in  this  thesis  to  explicitly 
determine  the  effectiveness  of  the  platform.  Second,  one  can  shop  by  features. 
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depending  on  the  written  catalog  descriptions  or  the  verbal  explanations  of 
vendors  to  provide  the  necessary  information  to  make  the  decision.  Third,  the 
trainer  can  conduct  a  user  study  in  which  a  representative  user  sample  compares 
several  similar  platforms  and  provides  comparative  subjective  evaluations. 
Fourth,  one  can  use  the  gut  instinct  of  what  looks  good  based  on  whatever 
experience  the  trainer  has  accumulated  in  the  domain. 

Based  on  this  thesis  project  as  a  whole,  we  conclude  that  the  training 
transfer  study  does  not  adequately  support  the  serious  game  acquisition  process. 
Quite  simply,  such  a  study  involves  much  effort  with  little  return.  The  process  of 
gaining  access  to  user  units,  developing  satisfactory  evaluation  mechanisms  that 
produce  scientific  results  without  handcuffing  the  unit’s  initiative,  and  following 
the  project  through  the  chaotic  demands  of  operational  military  life  requires  much 
effort,  readjustment  from  missed  opportunities,  and  sheer  luck.  Furthermore,  the 
results  tend  to  be  murky  at  best.  For  transfer  studies  in  the  group  training 
domain,  even  the  most  rigid  transfer  studies  leave  gaping  holes  for  the  skeptic  to 
launch  unanswerable  concerns. 

The  reader  should  not  interpret  the  preceding  paragraph  as  a  suggestion 
to  discard  training  transfer  studies  altogether.  Training  transfer  studies 
potentially  offer  unique  and  valuable  insight  to  the  research  community  that  no 
other  methodologies  can  attain.  If  nothing  else,  the  process  of  the  research 
project  itself  results  in  a  well-documented  investigation  into  the  specific  areas  of 
success  and  failure  of  the  training  tool.  However,  the  slow,  arduous,  and 
potentially  inconclusive  world  of  training  transfer  studies  will  most  likely  only  offer 
frustration  to  the  decision  maker  trying  to  spend  Department  of  Defense  dollars 
on  a  narrowly  defined  budget,  time  schedule,  and  performance  objective  set. 

The  military  trainer  must  decide  what  systems  to  buy  somehow,  and  the 
preceding  argument  only  serves  to  nay-say  one  method.  We  did  not  structure 
this  thesis  project  in  any  way  to  answer  the  question  at  hand,  but  we  feel  the 
lengthy  hands-on  effort  has  provided  ample  qualification  to  include  some  notes  of 

opinion  on  the  topic.  We  start  the  discussion  by  stating  up  front  that  the  answer 
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is  not  some  clean-cut  checklist  style  formula.  Rather,  a  sound  serious  game 
acquisition  effort  will  most  likely  include  some  aspect  of  each  of  the  last  three 
selection  methods. 

Arguably,  feature  shopping  drives  the  current  serious  game  acquisition 
process.  One  simply  jots  down  requirements,  reviews  vendor  descriptions  of 
applicable  products,  and  decides  according  to  some  designated  cost  analysis 
criteria.  The  Interservice/Industry,  Training,  Simulation,  and  Education 
Conference  (l/ITSEC)  offers  one  of  the  best  opportunities  for  the  military  trainer 
to  feature  shop.  Vendors  shamelessly  showcase  their  most  innovative 
contributions  to  the  simulation  community  with  flashy  screens,  fancy  interfaces, 
and  brilliant  sound  effects.  We  are  not  saying  l/ITSEC  is  bad;  quite  the  contrary, 
l/ITSEC  plays  an  absolutely  vital  role  in  educating  the  military  training  community 
about  the  available  market.  Nevertheless,  the  military  professional  who  treats 
l/ITSEC  as  a  buffet  is  doomed  to  come  home  with  an  empty  pocket  and  a 
warehouse  of  gadgets  that  will  gather  dust.  Common  sense  dictates  that  only  a 
foolish  buyer  relies  exclusively  on  the  advice  of  sales  professionals.  One  must 
shop  intelligently.  Feature  shopping  is  certainly  an  important  part  of  the  process, 
but  this  sort  of  selection  method  executed  exclusively  will  likely  result  in 
overpriced  bells  and  whistles  and  a  deficit  of  substance. 

We  illustrate  the  idea  of  user  studies  by  visiting  the  Marine  Corps’  recent 
efforts  to  update  the  individual  fighting  gear  for  Marines.  Over  the  past  decade 
and  a  half,  the  Marine  Corps  has  updated  virtually  all  of  the  personal  gear 
inventory,  including  packs,  sleeping  bags,  foul  weather  gear,  the  utility  uniform, 
and  boots.  For  the  most  part.  Marines  contentedly  use  this  gear,  happy  to  have 
emerged  from  the  outdated  world  of  shelter  halves  and  rubber  rain  suits.  The 
acquisition  program  employed  a  notable  technique  of  user  testing  for  all  of  this 
gear.  Representative  Marine  units  tested  the  gear  in  all  sorts  of  conditions,  and 
the  opinions  of  warfighting  Marines  literally  determined  the  course  of  the 
acquisition  effort.  This  technique  produced  high  quality  upgrades  that  have 
resulted  in  improvements  to  warfighting  capability  and  individual  morale. 
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Certainly,  this  technique  suits  the  acquisition  of  serious  games  as  well  with  one 
striking  exception.  A  Marine  testing  a  sleeping  bag  knows  that  he  needs  a 
product  that  is  light,  dry,  and  warm.  He  knows  this  from  his  experience  with  other 
sleeping  bags.  If  the  Marine  had  never  used  a  sleeping  bag  before,  he  might 
pick  a  light  sleeping  bag  that  left  him  wet  in  a  rainstorm  or  the  warmest  sleeping 
bag  that  weighed  too  much  for  practical  transport.  The  Marine’s  experience  with 
similar  products  enables  him  to  successfully  contribute  to  the  selection  process. 
In  the  case  of  serious  games,  most  Marines  do  not  have  the  requisite  experience 
to  know  what  they  want  or  to  know  when  the  product  is  “good.”  During  this 
project,  we  noticed  that  Marines  often  judged  VBS  2^^  training  either  because 
they  liked  the  novel  approach  or  because  the  non-traditional  approach  disturbed 
them.  Either  way,  serious  games  for  training  fell  outside  their  experience  base, 
leaving  them  unqualified  to  comment  on  the  quality  of  the  specific  platform. 

Military  decision  making  does  not  always  occur  with  a  highly  structured 
analytical  approach  due  to  time  constraints,  lack  of  resources,  or  other  obstacles. 
In  the  absence  of  structured  analysis,  military  professionals  often  make  quick 
decisions  based  on  available  information  and  relevant  experience.  The 
expression  “a  good  seventy  percent  solution  now  is  better  than  a  perfect  one 
hundred  percent  solution  later”  sums  up  this  concept.  Because  military  leaders 
employ  this  technique  in  training  and  war,  one  would  not  be  surprised  to  see 
them  employ  it  in  acquisition.  We  do  not  dismiss  this  technique  as  inappropriate 
to  the  acquisition  of  serious  games.  Certainly,  the  instantaneous  impression  of  a 
General  or  Master  Gunnery  Sergeant  with  decades  of  experience  can  lend  more 
to  the  selection  process  in  less  than  sixty  seconds  than  months  of  analysis. 
However,  such  a  process  employed  exclusively  has  neither  the  analytical  power 
required  by  those  holding  the  purse  strings  nor  the  breadth  to  represent  every 
user  situation.  At  its  worst,  this  technique  turns  into  a  single  individual  buying  pet 
toys,  and  at  its  best,  the  method  results  in  rapid  turn  around  that  catches  the  user 
base  offguard  and  leaves  critical  people  excluded  from  the  process. 
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For  serious  game  acquisition,  this  discussion  leaves  the  warfighter  in  a 
quandary.  If  the  complexities  of  measuring  team  cognition  preclude  a  bona  fide 
effectiveness  study,  the  user  must  buy  systems  intelligently  some  other  way.  In 
Section  1,  we  discussed  techniques  of  making  VBS  2™  ubiquitous  through  the 
Marine  Corps’  system  of  schools.  The  solution  to  the  serious  game  acquisition 
effort  lies  here  as  well.  This  thesis  demonstrates  that  serious  games  can 
effectively  contribute  to  training.  Schools  provide  the  means  to  quickly  get 
Marines  experienced  in  the  employment  of  serious  games  for  training.  Once 
exposed  to  serious  games  and  taught  how  to  use  them.  Marines  will  develop  the 
requisite  experience  base  to  support  the  user  studies  and  gut  instinct  buying 
approaches  that  can  best  select  the  most  suitable  training  platform  out  of  all  the 
competing  products  on  the  market. 

6.  Conservative  Use  of  Options 

VBS  2™  allows  the  user  to  manipulate  a  variety  of  options  to  enhance  the 
user  experience.  In  this  way,  one  can  control  whether  a  participant  dies,  how  the 
participants'  interface  works,  and  a  variety  of  other  practical  issues.  We  felt  that 
we  should  alter  these  options  from  the  defaults  with  caution.  The  Marine  Corps 
intends  to  make  VBS  2™  a  ubiquitous  training  asset,  and  Marines  will  expect  the 
training  device  to  look  the  same  and  operate  the  same  from  exercise  to  exercise 
and  from  command  to  command.  To  some  extent,  personalizing  training  assets 
allows  a  Marine  to  get  the  most  out  of  the  gear,  but  in  an  environment  of  constant 
turbulence  and  rotation,  there  is  value  in  all  assets  being  the  same.  We  did  not 
rule  out  the  idea  of  tailoring  the  VBS  2™  options  to  best  support  our  goals,  but 
we  took  a  conservative  approach  to  changing  them. 

After  conducting  several  training  exercises  in  the  series  of  pilot  studies 
and  experiments,  we  noted  some  options  that  we  felt  necessary  to  change  from 
the  default.  In  interface  training,  it  is  helpful  to  force  all  avatars  to  stay  alive 
regardless  of  battlefield  damage.  Despite  all  attempts  at  discipline  and  caution, 
novice  users  will  accidentally  shoot  each  other.  The  cost  in  terms  of  time  and 
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frustration  of  bringing  users  back  into  the  environment  is  not  worth  the  realism  for 
initial  training.  Of  course,  once  initial  training  is  done,  this  argument  does  not 
hold.  For  most  training  exercises,  invincibility  would  almost  certainly  teach 
Marines  the  wrong  thing.  Invincibility  is  only  useful  as  a  matter  of  convenience 
when  teaching  button  pushes. 

For  the  training  we  conducted,  it  was  helpful  to  disable  the  third  person 
view.  Likewise,  the  "M"  key  allows  users  to  pull  up  the  two-dimensional  map 
view.  While  the  map  is  quite  useful,  it  depicts  enemy  locations,  so  it  is  an 
unrealistic  asset.  We  disabled  the  "M"  key  function  so  that  users  could  not  see 
this  view.  In  general,  we  felt  that  the  best  training  evolved  from  users  only  seeing 
the  world  from  the  first-person  view  of  their  position.  With  this  in  mind,  we  paid 
attention  to  disable  all  other  views  and  to  position  participants  so  that  they  could 
only  use  their  own  computer  screens  to  see  the  environment. 

VBS  2™  has  a  re-spawn  capability  for  avatars  that  die  during  the  course 
of  the  scenario.  Avatars  re-spawn  into  swallows  that  can  fly  around  the  local 
area  of  their  unit  observing  the  action  from  a  third  person  view.  We  enabled  this 
function.  We  found  that  users  who  died  in  the  scenario  and  were  doomed  to 
stare  at  a  black  screen  for  the  remainder  of  the  exercise  got  little  training  and,  in 
some  instances,  detracted  from  the  training  of  others.  The  re-spawn  function  is 
quite  helpful  to  remedy  this  training  problem. 

7.  Not  All  Terrains  Are  Created  Equal 

During  the  course  of  our  experiments,  we  noticed  a  difference  between 
the  quality  of  the  Twentynine  Palms  terrain  and  the  Sahrani  terrain.  The  two 
terrain  databases  come  from  different  origins.  The  Twentynine  Palms  database 
hails  from  Marine  Corps  specifications,  while  the  Sahrani  database  satisfies  the 
needs  and  desires  of  the  gaming  community.  However,  the  Twentynine  Palms 
terrain  did  not  offer  the  same  quality  of  training  experience  as  the  Sahrani  terrain. 
Vehicles  in  the  Twentynine  Palms  terrain  maneuvered  on  roads  as  though  they 
drove  on  paved  highways  instead  of  the  off  road  trails  that  exist  in  reality. 
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Vehicles  in  the  Twentynine  Palms  terrain  could  easily  climb  mountains  that  they 
could  not  drive  on  in  reality.  The  microterrain  features  that  characterize  the 
Twentynine  Palms  landscape,  limiting  wheeled  mobility,  were  not  represented  at 
all.  Twentynine  Palms  is  full  of  rugged  rocks,  wadis,  holes,  craters,  and  other 
features  that  make  wheeled  movement  difficult,  or  sometimes  impossible,  and 
wheeled  vehicles  could  move  easily  across  open  terrain  in  the  virtual 
environment  database.  The  Sahrani  terrain,  on  the  other  hand,  provided  more 
realistic  roads,  hills,  microterrain  features,  and  other  features.  One  can  debate 
the  level  of  fidelity  needed  for  useful  training,  but  fidelity  must  certainly  be  high 
when  it  relates  to  the  specific  task  being  trained.  Convoy  training  revolves 
around  the  mobility  of  forces,  and  terrain  characteristics  dramatically  impact  this 
mobility.  The  unit  seeking  to  use  Twentynine  Palms  terrain  to  support  a 
rehearsal  scenario  could  be  sorely  disappointed.  The  unit  could  very  well  find 
that  vehicles  could  not  execute  a  well  planned  operation  because  of  the  realities 
of  the  ground. 

In  this  thesis,  we  do  not  specifically  seek  to  define  which  terrain  databases 
are  suitable  and  which  are  not.  We  feel  it  is  important  that  those  involved  in  the 
development  and  procurement  of  terrain  databases  for  VBS  evaluate  the 
objectives  of  the  terrain.  These  people  must  ensure  they  write  the  specifications 
for  VBS  2™  terrain  to  support  the  potential  associated  training  objectives.  Desert 
terrain  can  be  as  simple  as  a  sand  colored  plane,  and  a  building  can  be  as 
simple  as  a  textured  cube.  However,  some  cubes  scattered  on  a  tan  plane  do 
not  necessarily  make  adequate  terrain  for  the  trainee.  On  the  other  hand,  the 
trainee  does  not  necessarily  need  to  see  each  blade  of  grass  blowing  in  the  wind 
or  the  interior  of  every  building  in  the  terrain.  The  answer  lies  somewhere 
between  the  two,  and  the  arbitrary  middle  point  is  a  distinct  function  of  potential 
training  objectives.  VBS  2™  terrain  database  construction  offers  a  very  real 
potential  for  waste  of  money,  either  in  the  purchase  of  terrain  that  offers 
unnecessary  realism  or  the  development  of  terrain  that  is  so  poor  in  quality  that  it 
is  never  used. 
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8.  Communications  in  VBS  2™ 

During  this  project,  we  experimented  with  two  software  applications  to 
support  communications.  We  used  JVTR  and  CNR  SIM.  We  found  that  while 
we  could  accomplish  communications  objectives  with  either,  we  considered  both 
unsatisfactory  for  the  needs  of  our  project.  Quite  simply,  communications  should 
be  as  seamless  as  all  other  aspects  of  VBS  2™.  Communications  should  be 
internal  to  VBS  2™  with  seams  between  vendor  production  transparent  to  the 
user.  Our  current  version  of  VBS  2™  has  five  icons  on  the  startup  screen  for 
functions  such  as  setup,  networking,  editing,  and  starting  a  scenario.  A  user 
should  have  an  additional  icon  for  communications.  From  this  icon,  the  user  and 
administrator  should  be  able  to  access  appropriate  buttons  and  toggles  and  then 
start  the  simulation.  The  amount  of  headaches  associated  with  JVTR  and  CNR 
SIM  are  not  consistent  with  the  rest  VBS  2™.  Our  work  in  this  project 
demonstrated  consistently  through  all  five  experiments  that  communications  is 
fundamental  to  the  training  experience  and  one  of  the  most  important  aspects  of 
the  exercise.  The  communications  support  for  VBS  2™  needs  immediate 
attention. 

9.  Computer  Glitches  in  VBS  2™ 

We  noticed  a  few  computer  glitches  in  VBS  2™  that  bear  mention.  First, 
our  work  in  the  interface  familiarization  pilot  study  discussed  in  Chapter  III 
revealed  that  I  ED  disarmament  disrupts  the  AAR  recording.  As  we  have 
mentioned  previously,  the  AAR  often  provides  the  bulk  of  the  training  value  for  a 
VBS  2™  exercise.  If  I  ED  disarmament  plays  a  critical  role  in  the  scenario,  this 
problem  could  limit  the  training  value  of  the  simulation  exercise.  Likewise,  we 
discussed  the  problem  of  a  driver  and  gunner  losing  control  of  their  avatars  in 
some  circumstances  in  networked  scenarios.  We  found  that  this  problem  could 
be  remedied  by  switching  into  map  view  and  back  out  again,  but  this  problem 
inhibits  training.  Worse,  it  usually  occurs  right  in  the  heat  of  some  action  that  is 
the  center  of  the  training  objective. 
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We  understand  that  computer  glitches  such  as  these  have  been  identified 
to  Bohemia  Interactive  and  will  be  fixed  in  later  editions  of  VBS  2™.  VBS 
offers  a  Web  site  forum  for  the  discussion  of  such  topics.  In  this  forum,  users 
from  every  sector  of  the  VBS  2^^  community  post  lessons  learned  and  interact 
with  the  VBS  2™  support  staff  to  come  up  with  the  best  possible  answers.  This 
support  forum  is  vital  to  the  ultimate  success  of  VBS  2™,  because  it  allows  the 
software  package  to  be  a  living  thing,  continuously  adapting  to  the  needs  of  the 
user.  However,  it  also  requires  the  user  to  stay  attuned  to  the  latest  information 
to  provide  the  best  possible  training  for  the  unit.  Many  VBS  2™  questions  have 
been  asked,  and  many  answers  are  known,  but  ensuring  the  Marine  with  the 
question  reaches  the  applicable  answer  can  be  a  daunting  task. 

10.  Training  Notes 

We  collected  a  few  miscellaneous  notes  that  we  feel  worthy  to  document 
for  anyone  seeking  to  follow  in  our  footsteps  working  with  VBS  2™  as  a  training 
platform.  The  trainer  must  consider  time.  The  simplicity  of  a  laptop  trainer  may 
tempt  one  to  believe  that  training  can  occur  on  a  dime.  This  is  not  true,  at  least 
initially.  We  found  that  twelve  computers  require  approximately  one  man  hour  to 
set  up  and  one  man  hour  start  up.  This  rule  of  thumb  can  be  circumvented  by 
training  the  Marines  using  the  system  to  do  the  set  up  and  start  up  procedures, 
but  this  requires  its  own  amount  of  time  and  potentially  subjects  the  equipment  to 
abuse  that  is  difficult  to  see  and  control.  Additionally,  even  the  most  dedicated 
Marine  can  only  train  with  VBS  2™  so  long  before  it  turns  into  a  game,  and  a 
monotonous  one  at  that.  Limited  training  times  seem  to  promote  the  best 
training. 

The  trainer  must  seriously  consider  how  to  ensure  trainees  take  the 
simulation  seriously.  It  does  not  take  much  for  VBS  2™  to  transform  from 
training  into  a  game.  However,  this  is  true  of  live  training,  or  any  other  training, 
as  well,  and  the  solution  to  the  problem,  whether  live  or  simulated,  is  leadership. 
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Without  the  proper  leadership  in  place,  the  trainer  will  find  that  negligent 
discharges,  fratricides,  and  other  practices  associated  with  games  rather  than 
training  will  dominate  the  exercise. 

VBS  2™  is  not  only  a  suitable  forum  for  observing  trainees,  but  it  provides 
an  opportunity  to  observe  the  trainer.  Often,  the  trainer  is  as  much  of  a  novice  at 
training  Marines  as  the  Marines  are  at  conducting  the  training.  For  the  Company 
Commander  seeking  insight  into  how  his  Platoon  Commander  trains  his  Marines 
or  for  the  Platoon  Commander  trying  to  show  his  new  Platoon  Sergeant  how  to 
train,  VBS  2™  provides  an  excellent  opportunity  to  discuss  what  is  important  and 
how  to  present  it  to  the  training  audience. 

C.  RECOMMENDATIONS  FOR  FUTURE  WORK 

1.  Repeat  Experiment  5  to  See  the  Effects  of  the  Trainer 

In  Experiment  5,  we  unsuccessfully  attempted  to  compare  simulation 
training  exercises  in  which  the  trainer  implemented  training  through  a  third  party 
with  exercises  administered  by  the  trainer  alone.  However,  our  observations 
provided  ample  evidence  that  this  domain  of  study  bears  further  investigation.  If 
a  researcher  could  convince  a  unit  to  train  Marines  in  different  venues,  a 
comparison  study  using  the  unit’s  DVTE  assets  versus  the  same  assets  in  a 
Simulation  Center  would  shed  light  on  this  matter.  We  believe  this  question  is 
highly  important,  because  it  drives  decisions  about  whether  the  Marine  Corps 
and  other  services  should  field  and  maintain  serious  games  through  operational 
units  or  only  house  and  administer  them  through  contractor  staffs  in  base 
Simulation  Centers. 

2.  Can  VBS  2™  Scenarios  be  Catalogued  and  Classified? 

We  noted  that  VBS  2™  could  move  toward  ubiquitousness  as  the  user 
base  develops  a  body  of  tried  and  tested  scenarios  to  pass  throughout  the  user 
community.  The  less  Marines  have  to  invent  the  wheel,  the  more  inclined  they 


196 


will  be  to  use  the  tool.  Right  now,  there  is  no  good  mechanism  to  pass  VBS  2™ 
scenarios  around  in  a  structured,  organized  way.  In  order  for  one  Marine  to  use 
another  Marine’s  scenario,  the  Marine  must  know  the  purpose  of  the  scenario, 
the  training  objectives  it  should  fill,  the  number  of  players  involved,  the  type  of 
support  involved,  and  a  variety  of  other  information.  JTCOIC  has  made  great 
strides  in  documenting  how  they  made  their  scenarios,  what  they  do,  and  what 
they  look  like  and  has  even  started  including  video  to  support  the  documentation. 
Future  work  could  investigate  Best  Practices  for  formalizing  such  documentation 
so  that  the  operator  could  select  VBS  2™  scenario  support  from  a  menu,  easily 
matching  scenario  characteristics  to  training  needs. 

3.  Do  Peripheral  Computer  Devices  Enhance  VBS  2™  Training? 

Currently,  the  Marine  Corps  fields  VBS  2™  with  the  idea  that  it  will  only  be 
used  on  a  laptop  with  the  standard  keyboard  and  mouse  interface.  One  can  alter 
the  interface  by  adding  game  controllers,  larger  screens,  head  mounted  displays, 
more  realistic  communications  interfaces,  and  other  peripheral  devices.  VBS  2™ 
derives  a  certain  amount  of  benefit  from  its  easy  deployability;  at  the  end  of  the 
day,  a  Marine  only  needs  to  move  around  a  laptop  and  some  cables.  Future 
work  could  investigate  the  addition  of  some  of  these  peripheral  devices  to 
determine  whether  they  add  enough  training  benefit  to  justify  their  cost  and 
added  logistical  burden. 

4.  Would  a  VBS  2™  Filming  Capability  Be  Useful  for  Training? 

VBS  2™  shows  potential  for  use  as  a  demonstration  tool  as  well  as  a 
practical  application  tool.  A  trainer  can  choreograph  an  exercise  with  relative 
ease  to  show  Marines  what  “right”  looks  like.  This  idea  is  not  novel;  the  Army 
used  DARWARS  Ambush!  in  this  mode  (Roberts,  Diller,  &  Schmitt,  2006),  and 
JTCOIC  has  produced  videos  to  demonstrate  its  products.  However,  one  cannot 
easily  film  VBS  2™  exercises  for  demonstration  purposes.  The  trainer  must 
figure  out  how  to  use  an  external  screen  capture  tool  and  must  do  all  video 
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editing  externally.  Future  work  could  investigate  whether  an  internal  VBS  2™ 
filming  function  could  provide  training  benefit  and  how  this  video  function  should 
be  added  to  the  software. 

5.  When  Is  the  Third-Person  View  Preferabie  to  the  Map  View  for 
the  Real-Time  Editor  and  After-Action  Review  Functions? 

We  noticed  that  different  trainers  prefer  different  views  for  using  the  real¬ 
time  editor  and  the  AAR  tool.  While  the  flexibility  is  convenient  for  different 
people  to  use  VBS  2™  in  their  own  way,  a  better  understanding  of  when  each 
view  is  most  useful  would  be  helpful  in  training  the  trainer  to  make  the  best  use  of 
the  simulation.  Not  only  must  the  trainer  know  which  view  is  best  to  personally 
look  at  the  situation  and  affect  training,  but  the  trainer  must  know  the  best  view  to 
show  different  battlefield  activities  when  debriefing  exercise  participants.  Future 
work  could  investigate  how  a  trainer  can  best  use  these  different  views  to  provide 
the  most  training  value. 

6.  Is  There  a  Potential  for  Civilian  Users  of  Armed  Assault  to 
Contribute  to  Marine  Corps  VBS  2™  Training  Efforts? 

In  Chapter  II,  we  provided  an  extensive  review  of  gaming  throughout 
history  highlighting  the  continuing  link  between  civilian  leisure  gaming  and 
military  training.  VBS  2™  has  a  rich  history  in  the  civilian  gaming  industry.  VBS 
2™  originated  from  the  Operation  Flashpoint  game  engine,  and  the  current 
civilian  version  of  the  game.  Armed  Assault,  is  relatively  interchangeable  with 
VBS  2™.  VBS  2™  can  use  models,  terrain,  scenarios,  and  scripts  developed  for 
Armed  Assault.  Likewise,  Armed  Assault  can  operate  with  VBS  2™  creations. 
The  Armed  Assault  community  is  a  vibrant  group  with  extensive  interaction.  Web 
sites,  such  as  www.armaholics.com,  allow  Armed  Assault  users  to  share  their 
products  and  benefit  from  each  other’s  experience.  Typically,  Armed  Assault 
users  seek  realism  in  their  gaming  and  might  have  an  appreciation  for  something 
that  is  very  real:  military  gaming.  For  example,  while  the  Marine  Corps  and 
other  DoD  agencies  have  worked  on  building  realistic  Afghan  terrain  databases, 
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Afghan  terrain  has  shown  up  on  www.armaholics.com.  Taken  to  the  extreme, 
civilian  Armed  Assault  players  could  present  the  ultimate  oppositional  force  by 
playing  Marines  or  soldiers  in  their  own  game.  Future  work  could  explore 
whether  the  military  could  leverage  the  civilian  gaming  community  for  mutually 
beneficial  products,  thereby  gaining  training  assets  for  free. 

7.  Who  Is  This  New  Computer  Generation  and  What  Do  They 
Expect? 

The  simulation  training  community  has  been  quick  to  tout  simulation  training 
as  a  way  to  capitalize  on  the  technical  expertise  of  the  youth  filling  our  junior 
enlisted  ranks,  who  have  never  lived  in  a  time  without  computers.  We  have 
discussed  the  idea  that  these  young  people  may  not  all  have  the  gaming  expertise 
or  appreciation  for  gaming  that  we  think  they  do.  Moreover,  this  appreciation  of 
technology  may  not  spread  equally  across  ranks,  age  groups,  military  services  or 
components,  or  job  specialties.  For  example,  a  young  communicator  reservist 
may  appreciate  serious  games  for  training  more  than  an  active  duty  infantryman 
regularly  training  in  the  field.  We  have  seen  enough  of  this  issue  discussed  first¬ 
hand  to  highly  recommend  a  demographic  mapping  of  the  Marine  Corps  in  terms 
of  computer  gaming  expertise  and  appreciation.  An  organized  pursuit  of  serious 
games  for  training’s  sake  hinges  on  an  understanding  of  the  people  who  will  use  it. 
We  cannot  properly  design  interfaces,  train  users,  write  manuals,  or  field  systems 
without  a  better  understanding  of  the  Marine  sitting  at  the  keyboard.  This  study 
has  the  potential  to  extend  far  beyond  the  bounds  of  serious  games,  as  the  DoD 
investigates  the  potential  of  unmanned  equipment  on  the  battlefield,  bringing 
serious  gaming  and  reality  in  dangerous  proximity  to  each  other. 
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APPENDIX  A.  INFORMED  CONSENT  (EXPERIMENT  1) 


Informed  Consent  Form 

Introduction.  You  are  invited  to  participate  in  a  research  study  entitled  Virtual  Battlespace  2  User 
Familiarity  Pilot  Study.  This  study  supports  a  project  to  compare  the  training  effectiveness  of  a 
first-person  shooter  simulation  with  traditional  tactical  training  methods  such  as  the  sand  table. 
This  pilot  study  tests  procedures  to  familiarize  users  with  the  simulation. 

Procedures.  The  simulation  familiarization  will  consist  of  the  following: 

Survey  to  better  understand  the  user’s  level  of  eomputer  expertise; 

Overview  brief  deseribing  the  projeet  and  the  pilot  study’s  purpose; 

User  interface  brief  describing  the  basic  functions  of  the  simulation; 

User  experimentation  period,  where  partieipants  will  be  able  to  freely  test  the 
ideas  just  presented; 

User  evaluation,  where  partieipants  will  eomplete  a  short  exereise  demonstrating 
their  skill  level  with  the  simulation. 

The  pilot  study  will  take  no  longer  than  the  50-minute  elass  period. 

Risks.  The  potential  risks  of  participating  in  this  study  are  not  greater  than  minimal  risk.  The 
study  involves  no  known  reasonably  foreseeable  risks  or  hazards  greater  than  those  encountered 
in  everyday  life. 

Benefits.  The  anticipated  benefit  from  this  study  is  gaining  insight  into  the  viability  of  first-person 
shooter  simulations  as  tactical  training  devices. 

Compensation.  No  tangible  compensation  will  be  given.  A  copy  of  the  research  results  will  be 
available  at  the  conclusion  of  the  experiment.  If  you  would  like  a  copy  of  the  results,  e-mail  Major 
Ben  Brown  at  bjbrown@nps.edu. 

Confidentiality  &  Privacy  Act.  Any  information  that  is  obtained  during  this  study  will  be  kept 
confidential  to  the  full  extent  permitted  by  law.  All  efforts,  within  reason,  will  be  made  to  keep 
your  personal  information  in  your  research  record  confidential  but  total  confidentiality  cannot  be 
guaranteed.  No  information  which  could  identify  a  participant  will  be  publicly  accessible. 
Records  of  participation  will  be  maintained  by  NPS  for  3  years,  after  which  they  will  be 
destroyed.  However,  it  is  possible  that  the  researcher  may  be  required  to  divulge  information 
obtained  in  the  course  of  this  research  to  the  subject’s  chain  of  command  or  other  legal  body. 
Voluntary  Nature  of  the  Study.  Participation  in  this  study  is  strictly  voluntary,  and  if  agreement 
to  participation  is  given,  it  can  be  withdrawn  at  any  time  without  prejudice. 

Points  of  Contact.  It  is  understood  that  should  any  questions  or  comments  arise  regarding  this 
project,  or  a  research  related  injury  is  received,  the  Principal  Investigator,  Dr.  William  J.  Becker, 
656-3963,  wibecker@nps.edu  should  be  contacted.  Any  other  questions  or  concerns  may  be 
addressed  to  the  Navy  Postgraduate  School.  IRB  Chair,  LCDR  Paul  O’Connor  ,  831-656-3864, 
peoconno@nps.edu. 

Statement  of  Consent.  I  have  read  the  information  provided  above.  I  have  been  given  the 
opportunity  to  ask  questions  and  all  the  questions  have  been  answered  to  my  satisfaction.  I  have 
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been  provided  a  copy  of  this  form  for  my  records  and  I  agree  to  participate  in  this  study.  I 
understand  that  by  agreeing  to  participate  in  this  research  and  signing  this  form,  I  do  not  waive 
any  of  my  legal  rights. 


Participant’s  Signature 


Date 


Researcher’s  Signature 


Date 
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APPENDIX  B.  IRB  REQUEST  (EXPERIMENT  1) 


William  J.  Becker,  Ph.D. 
MOVES  Institute 
Watkins  Hall 

Naval  Postgraduate  School 
Monterey,  California  93943 


831-656-3963 
DSN:  756-3963 
Fax:  831-656-7599 
wjbecker@nps.edu 


To:  Protection  of  Human  Subjects  Committee 

Subject:  Application  for  Human  Subjects  Review  (Title):  Virtual  Battlespace  2  User  Familiarity 
Pilot  Study 

Projected  Start  Date:  02  /  16  /  2009 

Month  Day  Year 


I  am  requesting  approval  of  the  attached  experimental  protocol.  The  following  documentation  is 

provided  in  support  of  my  application. 

The  Principal  Investigator  understands  and  accepts  the  following 
obligations  to  protect  the  rights  and  welfare  of  research  subjects  in  this  study: 

•  I  recognize  that  as  the  Principal  Investigator  it  is  my  responsibility  to  ensure  that  this  research 
and  the  actions  of  all  project  personnel  involved  in  conducting  this  study  will  conform  with  the 
IRB  approved  protocol  and  IRB  requirements/policies. 

•  I  recognize  that  it  is  my  responsibility  to  ensure  that  valid  informed  consent  /  assent  (unless 
explicitly  waived  by  the  IRB)  has  been  obtained  from  all  research  subjects  or  their  legally 
authorized  representatives.  I  will  ensure  that  all  project  personnel  involved  in  the  process  of 
consent  are  trained  properly  and  are  fully  aware  of  their  responsibilities  relative  to  obtaining 
informed  consent  /  assent  according  to  the  IRB  guidelines. 

•  I  will  ensure  all  personnel  involved  in  this  study  have  completed  the  required  IRB  Training. 

•  I  will  not  initiate  any  change  in  protocol  without  IRB  approval. 

•  I  have  no  conflict  of  interest  negating  me  from  performing  this  research. 

•  I  will  maintain  all  required  research  records  on  file;  and  I  recognize  that  the  IRB  is  authorized 
to  inspect  these  records  at  any  time. 

•  I  will  immediately  inform  the  IRB  Chair  and  NPS  Dean  of  Research  of  any  untoward  event  or 
injury  that  involves  a  research  participant. 

•  I  understand  that  in  the  absence  of  a  continuing  review  and  approval,  this  research  may  not 
continue  beyond  the  end  of  the  approval  period. 

•  At  the  completion  of  this  project,  an  End-of-Experiment  Report  will  be  submitted. 

•  I  will  not  commence  this  research,  including  subject  recruitment,  until  I  have  received  my  NPS 
IRB  application  approval  letter. 


Dr,  William  J.  Becker 


(Signature  of  Principal  Investigator) 
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Application  for  Human  Subjects  Review  NFS  IRB  Number: 


Principal  Investigator(s): 
Co-  PI(s) 


Dr.  William  J.  Becker,  Professor,  831-656-3963 
Major  Benjamin  J.  Brown,  Student,  831-656-3812 


Title  of  Experiment:  Virtual  Battlespaee2  User  Familiarity  Pilot  Study 


Approval  Requested 


[  X  ]  New 


[  ]  Continuing  [  ]  Amendment 


Requested  Level  of  Risk  [  ]  Exempt  [  X  ]  Minimal  [  ]  More  than  Minimal 
Justifieation;  The  participants  will  be  asked  to  use  virtual  training  simulation  on  laptop  computers, 
whieh  is  a  standard  practice  for  this  currieulum. _ 


Work  to  be  done  in  (Site/Bldg/Rm) 
Watkins  275  /  285 

Estimated  date  of  completion  (not  to  exeeed  one  year  from 
start  date):  30  Sept  2009 

Maximum  number  of  subjects: 

50 

Estimated  length  of  each  subjects  participation: 

50  minutes 

Special  Populations  that  will  be  Used  as  Participants: 


[  ]  Subordinates  [  ]  Minors  [  X  ]  NPS  Students  [  ]  Special  Needs  (e.g.  Pregnant  women) 

Speeify  safeguards  to  avoid  undue  influence  and  protect  subject’s  rights: 

Participation  is  voluntary,  and  each  participant  will  have  the  option  to  withdraw  from  the 
study  at  any  time. 

Partieipants  will  be  informed  that  class  credit  is  not  involved  in  the  study  either  for 
participation  or  for  lack  of  it. 

The  data  collected  in  the  study  will  be  safeguarded  as  requested  by  generally  aecepted  IRB 
standards:  each  person  will  be  identified  only  as  a  eode  number  on  all  research  forms  and 
databases;  the  name  of  any  person  on  any  signed  doeument  will  not  be  paired  with  the  code 
number  in  order  to  proteet  identity;  and  records  of  the  partieipant’s  participation  will  be 

_ maintained  by  NPS  for  3  years,  after  which  they  will  be  destroyed. _ 

Scientific  Merit  Review  (Cheek  all  that  apply) 

[  X  ]  This  research  is  part  of  a  funded  project  (Job  Order  Number:  R9554  ) 

[  ]  This  researeh  is  a  student  thesis  (Attaeh  a  copy  of  the  approved  thesis  proposal) 

[  ]  Other  (Attach  a  complete  research  proposal  -  Dept.  Chair  must  sign  Application  Cover  Letter) 

Outside  Cooperating  Investigators  and  Ageneies: 

None 

[  ]  A  copy  of  the  cooperating  institution’s  POC  and  CO’s  approval  is  attached. _ 

Description  of  Research:  (attach  an  additional  sheet  if  needed).  The  purpose  of  the  pilot  study  is  to 
test  a  proeedure  to  familiarize  users  with  Virtual  Battlespaee  2  {VBS  2).  VBS  2  is  a  personal 
eomputer-based  first-person  shooter  simulation  used  by  the  Marine  Corps.  The  simulation  is 
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installed  under  eontraet  on  Dell  XPS  eomputers  and  deployed  in  suites  of  16  to  operational  infantry 
units.  NPS  has  6  sueh  eomputers,  and  an  additional  suite  of  16  have  been  obtained  for  this  projeet. 
These  eomputers  will  be  used  for  the  pilot  study.  The  pilot  study  antieipates  a  training  transfer 
study  of  Virtual  Battlespaee  2  using  a  platoon-level  eonvoy  exereise. 

Simulation  familiarization  will  eonsist  of  the  following: 

Survey  to  better  understand  the  user’s  level  of  eomputer  expertise; 

Overview  brief  deseribing  the  projeet  and  the  pilot  study’s  purpose; 

User  interfaee  brief  deseribing  the  basie  funetions  of  the  simulation; 

User  experimentation  period,  where  partieipants  will  be  able  to  freely  test  the  ideas  just 
presented; 

User  evaluation,  where  partieipants  will  eomplete  a  short  exereise  demonstrating  their  skill 
level  with  the  simulation. 

For  the  study,  students  will  be  randomly  assigned  to  groups  of  3  to  partieipate  as  a  HMMWV  erew 
in  the  positions  of  vehiele  eommander,  driver,  and  gunner.  Tasks  to  be  evaluated  inelude  driving, 
navigating,  mounting  and  dismounting  the  vehiele,  employing  the  vehiele’s  maehine  gun,  and 
employing  personal  weapons.  The  pilot  study  will  take  no  longer  than  a  50-minute  elass  period. 
Method  of  Subjeet  Reeruitment:  (attaeh  an  additional  sheet  if  needed).  Professor  John  Falby  has 
offered  to  solieit  his  MV3922  elass  for  volunteers  for  the  pilot  study.  The  elass  is  a  seminar 
environment,  so  students  will  have  no  pressure  to  partieipate  in  the  study  for  elass  eredit. 
Additional  volunteers  will  be  reeruited  by  e-mail  from  the  remaining  pool  of  MOVES  students. 


I  have  read  and  understand  NPS  poliey  on  the  Proteetion  of  Human  Subjeets.  If  there  are  any 
ehanges  in  any  of  the  above  information  or  any  ehanges  to  the  attaehed  materials,  I  will  suspend  the 
experiment  until  I  obtain  new  IRB  approval. 

SIGNATURE _  DATE _ 

SIGNATURE  DATE 
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APPENDIX  C.  INFORMED  CONSENT  (EXPERIMENTS  2  AND  3) 


Informed  Consent  Form 

Introduction.  You  are  invited  to  participate  in  a  research  study  entitled  Virtual  Battlespace  2 
Scenario  Testing  Pilot  Study.  This  study  supports  a  project  to  compare  the  training  effectiveness  of 
a  first-person  shooter  simulation  with  traditional  tactical  training  methods  such  as  the  sand  table. 
This  pilot  study  tests  computer  simulation  scenarios  used  for  the  tactical  training. 

Procedures.  The  simulation  familiarization  will  consist  of  the  following: 

Survey  to  better  understand  the  user’s  level  of  eomputer  expertise,  relevant 
taetieal  experienee,  and  attitudes  toward  simulation  for  tactieal  training; 

Overview  brief  deseribing  the  project  and  the  pilot  study’s  purpose; 

User  interfaee  brief  deseribing  the  basie  funetions  of  the  simulation; 

User  experimentation  period,  where  partieipants  will  be  able  to  freely  test  the 
ideas  just  presented; 

User  evaluation,  where  partieipants  will  complete  a  short  exereise  demonstrating 
their  skill  level  with  the  simulation. 

Overview  brief  of  taeties,  teehniques,  and  proeedures  relevant  to  the  eomputer 
exereise. 

Simulation  exereise  to  train  small  unit  eonvoy  immediate  action  drills. 

Simulation  exercise  to  test  small  unit  eonvoy  immediate  aetion  drills. 

Surveys  to  determine  the  user’s  impression  of  the  simulation  training. 

The  pilot  study  will  take  no  longer  than  one  training  day  (8  hours). 

Risks.  The  potential  risks  of  participating  in  this  study  are  not  greater  than  minimal  risk.  The 
study  involves  no  known  reasonably  foreseeable  risks  or  hazards  greater  than  those  encountered 
in  everyday  life. 

Benefits.  The  anticipated  benefit  from  this  study  is  gaining  insight  into  the  viability  of  first-person 
shooter  simulations  as  tactical  training  devices. 

Compensation.  No  tangible  compensation  will  be  given.  A  copy  of  the  research  results  will  be 
available  at  the  conclusion  of  the  experiment.  If  you  would  like  a  copy  of  the  results,  e-mail  Major 
Ben  Brown  at  bjbrown@nps.edu. 

Confidentiality  &  Privacy  Act.  Any  information  that  is  obtained  during  this  study  will  be  kept 
confidential  to  the  full  extent  permitted  by  law.  Simulation  sessions  will  be  recorded  using  the 
system  After  Action  Review  tool,  but  the  recording  only  shows  generic  avatar  representations  of 
participants  thereby  guaranteeing  anonymity.  All  efforts,  within  reason,  will  be  made  to  keep 
your  personal  information  in  your  research  record  confidential  but  total  confidentiality  cannot  be 
guaranteed.  No  information  which  could  identify  a  participant  will  be  publicly  accessible. 
Records  of  participation  will  be  maintained  by  NPS  for  3  years,  after  which  they  will  be 
destroyed.  However,  it  is  possible  that  the  researcher  may  be  required  to  divulge  information 
obtained  in  the  course  of  this  research  to  the  subject’s  chain  of  command  or  other  legal  body. 
Voluntary  Nature  of  the  Study.  Participation  in  this  study  is  strictly  voluntary,  and  if  agreement 
to  participation  is  given,  it  can  be  withdrawn  at  any  time  without  prejudice. 
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Points  of  Contact.  It  is  understood  that  should  any  questions  or  comments  arise  regarding  this 
project,  or  a  research  related  injury  is  received,  the  Principal  Investigator,  Dr.  William  J.  Becker, 
831-656-3963,  wibecker@, nps.edu  should  be  contacted.  Any  other  questions  or  concerns  may  be 
addressed  to  the  Navy  Postgraduate  School.  IRB  Chair,  LCDR  Paul  O’Connor  ,  831-656-3864, 
peoconno@nps.edu. 

Statement  of  Consent.  I  have  read  the  information  provided  above.  I  have  been  given  the 
opportunity  to  ask  questions  and  all  the  questions  have  been  answered  to  my  satisfaction.  I  have 
been  provided  a  copy  of  this  form  for  my  records  and  I  agree  to  participate  in  this  study.  I 
understand  that  by  agreeing  to  participate  in  this  research  and  signing  this  form,  I  do  not  waive 
any  of  my  legal  rights. 


Participant’s  Signature 


Date 


Researcher’s  Signature 


Date 
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APPENDIX  D.  IRB  REQUEST  (EXPERIMENTS  2  AND  3) 


William  J.  Becker,  Ph.D. 
MOVES  Institute 
Watkins  Hall 

Naval  Postgraduate  School 
Monterey,  California  93943 


wjbecker@nps.edu 


Fax:  831-656-7599 


831-656-3963 
DSN:  756-3963 


To:  Protection  of  Human  Subjects  Committee 


Subject:  Application  for  Human  Subjects  Review  (Title):  Virtual  Battlespace  2  Scenario 
Testing  Pilot  Study 

Projected  Start  Date:  04  /  16  /  2009 


Month  Day 


Year 


I  am  requesting  approval  of  the  attached  experimental  protocol.  The  following  documentation  is 

provided  in  support  of  my  application. 

The  Principal  Investigator  understands  and  accepts  the  following 
obligations  to  protect  the  rights  and  welfare  of  research  subjects  in  this  study: 

•  I  recognize  that  as  the  Principal  Investigator  it  is  my  responsibility  to  ensure  that  this  research 
and  the  actions  of  all  project  personnel  involved  in  conducting  this  study  will  conform  with  the 
IRB  approved  protocol  and  IRB  requirements/policies. 

•  I  recognize  that  it  is  my  responsibility  to  ensure  that  valid  informed  consent  /  assent  (unless 
explicitly  waived  by  the  IRB)  has  been  obtained  from  all  research  subjects  or  their  legally 
authorized  representatives.  I  will  ensure  that  all  project  personnel  involved  in  the  process  of 
consent  are  trained  properly  and  are  fully  aware  of  their  responsibilities  relative  to  obtaining 
informed  consent  /  assent  according  to  the  IRB  guidelines. 

•  I  will  ensure  all  personnel  involved  in  this  study  have  completed  the  required  IRB  Training. 

•  I  will  not  initiate  any  change  in  protocol  without  IRB  approval. 

•  I  have  no  conflict  of  interest  negating  me  from  performing  this  research. 

•  I  will  maintain  all  required  research  records  on  file;  and  I  recognize  that  the  IRB  is  authorized 
to  inspect  these  records  at  any  time. 

•  I  will  immediately  inform  the  IRB  Chair  and  NPS  Dean  of  Research  of  any  untoward  event  or 
injury  that  involves  a  research  participant. 

•  I  understand  that  in  the  absence  of  a  continuing  review  and  approval,  this  research  may  not 
continue  beyond  the  end  of  the  approval  period. 

•  At  the  completion  of  this  project,  an  End-of-Experiment  Report  will  be  submitted. 

•  I  will  not  commence  this  research,  including  subject  recruitment,  until  I  have  received  my  NPS 
IRB  application  approval  letter. 


(Signature  of  Principal  Investigator) 


Dr,  William  J.  Becker 
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Application  for  Human  Subjects  Review  NFS  IRB  Number: 


Principal  Investigator(s): 
Co-  PI(s) 


Dr.  William  J.  Becker,  Professor,  831-656-3963 
Major  Benjamin  J.  Brown,  Student,  831-656-3812 


Title  of  Experiment:  Virtual  Battlespaee2  Scenario  Testing  Pilot  Study 


Approval  Requested 


[  X  ]  New 


[  ]  Continuing  [  ]  Amendment 


Requested  Level  of  Risk  [  ]  Exempt  [  X  ]  Minimal  [  ]  More  than  Minimal 
Justifieation;  The  participants  will  be  asked  to  use  virtual  training  simulation  on  laptop  computers, 
whieh  is  a  standard  practice  for  this  currieulum. _ 


Work  to  be  done  in  (Site/Bldg/Rm) 
Unit’s  elassroom  on  site 

Estimated  date  of  completion  (not  to  exeeed  one  year  from 
start  date):  31  Dec  2009 

Maximum  number  of  subjects: 

100 

Estimated  length  of  each  subjects  participation: 

8  hours 

Special  Populations  that  will  be  Used  as  Participants: 

[  ]  Subordinates  [  ]  Minors  [  ]  NPS  Students  [  ]  Speeial  Needs  (e.g.  Pregnant  women) 

Speeify  safeguards  to  avoid  undue  influence  and  protect  subject’s  rights: 

Participation  is  voluntary,  and  each  participant  will  have  the  option  to  withdraw  from  the 
study  at  any  time. 

Partieipants  will  be  informed  that  class  credit  is  not  involved  in  the  study  either  for 
participation  or  for  lack  of  it. 

The  data  collected  in  the  study  will  be  safeguarded  as  requested  by  generally  aecepted  IRB 
standards:  each  person  will  be  identified  only  as  a  eode  number  on  all  research  forms  and 
databases;  the  name  of  any  person  on  any  signed  doeument  will  not  be  paired  with  the  code 
number  in  order  to  protect  identity;  and  records  of  the  partieipant’s  participation  will  be 
maintained  by  NPS  for  3  years,  after  whieh  they  will  be  destroyed. 

The  simulation  system  After  Action  Review  tool  will  be  used  to  reeord  simulation  sessions. 

_ The  tool  uses  generie  avatar  representations  of  participants  thereby  guaranteeing  anonymity. 

Scientific  Merit  Review  (Cheek  all  that  apply) 

[  X  ]  This  researeh  is  part  of  a  funded  project  (Job  Order  Number:  R9554  ) 

[  ]  This  research  is  a  student  thesis  (Attach  a  copy  of  the  approved  thesis  proposal) 

[  ]  Other  (Attach  a  complete  researeh  proposal  -  Dept.  Chair  must  sign  Applieation  Cover  Letter) 


Outside  Cooperating  Investigators  and  Agencies: 

None 

[  ]  A  copy  of  the  eooperating  institution’s  POC  and  CO’s  approval  is  attaehed. 
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Description  of  Research:  (attach  an  additional  sheet  if  needed).  The  purpose  of  the  pilot  study  is  to 
test  small  unit  tactical  convoy  training  scenarios  in  Virtual  Battlespace  2  {VBS  2).  VBS  2  is  a 
personal  computer-based  first-person  shooter  simulation  used  by  the  Marine  Corps.  The  simulation 
is  installed  under  contract  on  Dell  XPS  computers  and  deployed  in  suites  of  32  to  operational 
infantry  units.  NPS  has  one  of  these  32  computer  suites  for  this  project.  These  computers  will  be 
used  for  the  pilot  study.  The  pilot  study  anticipates  a  training  transfer  study  of  Virtual  Battlespace 
2  using  a  platoon-level  convoy  exercise. 

Simulation  familiarization  will  consist  of  the  following: 

Survey  to  better  understand  the  user’s  level  of  computer  expertise,  relevant  tactical 
experience,  and  attitudes  toward  simulation  for  tactical  training; 

Overview  brief  describing  the  project  and  the  pilot  study’s  purpose; 

User  interface  brief  describing  the  basic  functions  of  the  simulation; 

User  experimentation  period,  where  participants  will  be  able  to  freely  test  the  ideas  just 
presented; 

User  evaluation,  where  participants  will  complete  a  short  exercise  demonstrating  their  skill 
level  with  the  simulation. 

Overview  brief  of  tactics,  techniques,  and  procedures  relevant  to  the  computer  exercise. 
Simulation  exercise  to  train  small  unit  convoy  immediate  action  drills. 

Simulation  exercise  to  test  small  unit  convoy  immediate  action  drills. 

Surveys  to  determine  the  user’s  impression  of  the  simulation  training. 

For  the  study.  Marines  will  be  assigned  to  crews  of  3  to  participate  as  a  HMMWV  crew  in  the 
positions  of  vehicle  commander,  driver,  and  gunner.  Vehicles  will  operate  in  convoys  of  5  to  9 
vehicles  The  pilot  study  evaluates  three  convoy  training  scenarios:  a  user  familiarity  scenario,  an 
immediate  action  drill  training  scenario,  and  a  computer  simulation  replication  of  a  live  Marine 
Corps  convoy  training  course.  The  study  uses  the  simulation’s  After  Action  Review  tool  and  user 
surveys  to  determine  the  effectiveness  of  the  training  scenarios  and  troubleshoot  scenario  design  for 

future  work.  The  pilot  study  will  take  no  longer  than  one  training  day. _ 

Method  of  Subject  Recruitment:  (attach  an  additional  sheet  if  needed).  A  Marine  Corps  unit  will  be 
solicited  for  volunteers  for  the  pilot  study.  Participant  solicitation  will  be  effected  through  the 
unit’s  leadership.  Active  duty  and  reserve  units  will  be  recruited  by  personal  contact  to  the 
command  staff  of  each  unit.  Command  staff  will  decide  who,  what,  and  where  units  will 
participate. _ 


I  have  read  and  understand  NPS  policy  on  the  Protection  of  Human  Subjects.  If  there  are  any 
changes  in  any  of  the  above  information  or  any  changes  to  the  attached  materials,  I  will  suspend  the 
experiment  until  I  obtain  new  IRB  approval. 

SIGNATURE _  DATE _ 

SIGNATURE  DATE 


211 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


212 


APPENDIX  E.  VBS  2™  CHEAT  SHEET  (EXPERIMENT  1) 


General  I 

Move  forward 

w  t 

Move  backward 

s  i 

Move  left 

A  ^ 

Move  right 

D  ^ 

Action  Menu 

[ 

INDIVIDUAL 

Run 

Shift  W 

Free  look 

L  Alt  +  L  Alt 

Lean  left 

Q 

Lean  right 

E 

Crouch 

X 
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z 

Stand 

c 

PERSONAL  GEAR 

Fire  weapon  /  Throw  grenade 

Left  click 

Cycle  weapons 

Space  bar 

Reload 

R 
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V 

Lower  /  Raise  weapon 

L  Ctrl  +  L  Ctrl 

Time 

T 

GPS 

R  Ctrl  +  M 

Night  vision 

N 

Binoculars 

B 

Compass 

G 

VEHICLE 

Interact  with  vehicle 

U 
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Q 

Forward,  fast  gear 

E 
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S  (Back  up) 

Get  out  of  the  vehicle 

H  +  H 

OBJECTS 

Recover  casualty 

[  +  “Carry  body” 

[  +  “Put  body  into 
vehicle” 

Recover  vehicle 

Get  in  front  (<  15  m) 

U  +  Tow  icon 

Disable  lED 

[  +  “Disarm  bomb” 
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APPENDIX  F.  VBS  2™  CHEAT  SHEET  (EXPERIMENTS  2  -  5) 
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OBJECTS 


Recover 

casualty 

(1) [  +  select  “Carry  body” 

(2)  [  +  select  “Put  body  Into  vehicle" 

Recover 

Vehicle 

(1)  Get  In  front  of  vehicle  <  15  m 

(2) 11  +  select  tow  Icon 
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APPENDIX  G.  FAMILIARITY  PILOT  STUDY  PREPARATION 


Computer  setup 
>  6  groups  of  2  computers: 

■  Connect  each  box  of  4  computers  together  on  a  table  with 
the  box’s  switch 


>  On  the  screen  of  each  computer  is  a  number:  1 1 3-X;  X  is  the 
computer  number 
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>  Use  5’  and  12’  cables  to  connect  to  switch 

>  Only  unwind  outlet  end  of  power  cords  for  each  computer 

>  Do  not  unwind  mouse  cord;  plug  in  as  is 

>  Use  one  power  strip  for  each  set  of  computers 

>  Start  all  computers  with  “Instructor”,  password  InstructDVTE 

■  Odd  computers:  Start  VTK  Admin 

■  Even  computers:  Start  VTK  User;  Fill  in  nickname  as 
computer  number 

>  All  computers  to  Networking 
Starting  the  scenario 

>  Odd  computers:  Go  to  “New”  and  click  on  “Familiarity” 

>  Even  computers:  Join  the  odd  computer’s  scenario  (make  sure 
it  is  the  computer  in  its  pair 

>  Odd  computers:  Click  OK  and  when  Even  comes  up  Continue 

>  Note  that  all  4  computers  will  show  up  together  with  two 
separate  Familiarity  scenarios 

Monitoring  the  scenario 

>  When  evaluation  starts,  start  recording  on  odd  computer  (ESC- 
Real  Time  Editor-Click  red  button  in  upper  right  corner) 

>  If  someone  gets  the  map,  they  clicked  the  M  key;  hit  ESC  to  get 
back  to  normal  user  view 

Cleanup 

>  SAVE  THE  AAR:  ESC-Real  Time  Editor-Click  red  button  in 
upper  right  corner 

■  Save  as  Pilot_Familiarity_X_Y  where  X  and  Y  are  the  two 
computer  numbers 

>  Ensure  a  survey  form  is  collected  for  each  computer 

>  Wind  up  all  cords 

>  Ensure  computers  go  back  to  proper  box 
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APPENDIX  H.  FAMILIARITY  PILOT  STUDY  QUESTIONNAIRE 

Please  fill  in  the  following  questionnaire.  All  information  will  be  held  confidential.  If  you 
need  to  expand  any  answer,  please  use  the  comments  section  on  the  reverse  side  of  your 
paper. 

1 .  Were  you  required  to  use  computer-based  simulations  as  part  of  your  training  in  the  past? 

NO /YES  (circle  one) 

If  YES,  what  simulations  have  you  used,  and  what  skills  were  they  used  to  train  (for  example, 
first-person  shooter,  flight  simulator,  operational  tactics,  etc.)? _ 

2.  Do  you  have  experience  playing  commercial  video  games? 

NO /YES  (circle  one) 

If  YES,  what  games  have  you  played,  and  what  kind  of  games  were  they  (for  example,  first- 
person  shooter,  flight  simulation,  sports,  fantasy,  etc.)? _ 

If  YES,  how  often  do  you  use  video  games? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

3.  Do  you  own  a  personal  computer? 

NO /YES  (circle  one) 

If  YES,  how  often  do  you  use  the  computer? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

4.  What  hand  do  you  use  to  operate  a  computer  mouse? 

LEFT  /  RIGHT  /  EITHER  (circle  one) 

5.  How  often  do  you  use  the  following  computer  applications?  (please  answer  all  that  apply) 

a.  E-mail:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

b.  Browse  web:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

c.  Video:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

d.  Music:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

e.  MS  Word:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

f.  MS  Other:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

6.  Participant  Number:  _ _ _ 

7.  Date:  _ 

8.  Year  of  birth:  _ 

9.  Service  component:  _ 

10.  Primary  MOS  or  job  specialty:  _ 

11.  Rank:  _ 

12.  Time  in  service:  _ years _ months  **  TURN  PAPER  OVER 
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13.  Circle  your  role  in  the  evaluation  exercise:  DRIVER  GUNNER  (circle  one) 

14.  How  many  targets  did  you  see  in  the  evaluation?  Personnel _  Vehicle _ _ 

15.  Without  consulting  the  cheat  sheet,  what  key  do  you  use  to  drive  a  vehicle  forward? _ 

16.  Briefly  describe  the  steps  to  tow  a  vehicle.  _ 


17.  What  task  was  hardest  for  you? 


18.  What  task  was  easiest  for  you? 


19.  Rate  your  confidence  in  doing  the  following  tasks  in  the  simulation  by  checking  one  block  for 
each  task  (1  means  you  are  NOT  confident;  5  means  you  are  HIGHLY  confident): 


Body  movements  (walking,  lying  in  prone,  etc.) 

Vehicle  movements 
Shooting  your  rifle 
Shooting  vehicle  mounted  machine  gun 
Recovering  a  casualty 
Recovering  a  vehicle  (towing) 

Disarming  an  lED 

20.  Do  you  have  any  suggestions  for  improving  user  familiarization  training? 


1 

2 

3 

4 

5 

Comments: 


Thank  you  for  volunteering  to  participate  in  this  study. 
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APPENDIX  I.  FAMILIARITY  FOLLOW  UP  EVALUATION 


Participant  number 

When  instructed  to  begin,  perform  the  following  tasks 
in  order: 

1 .  Shoot  the  enemy  combatant. 

2.  Board  HMMWV 

3.  Drive  to  enemy  combatant. 

4.  Dismount  HMMWV. 

5.  Load  enemy  combatant’s  body  into  HMMWV. 

6.  Engage  truck  target  with  grenade. 

7.  Assume  prone  position. 

8.  Engage  near  Echo  silhouette  target  in  semi¬ 
automatic  rifle  mode. 

9.  Engage  far  Echo  silhouette  target  in  semi¬ 
automatic  rifle  mode. 

10.  Stand  up. 

1 1 .  Use  simulation  watch  and  write  time: 


221 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


222 


APPENDIX  J.  MOVES  PILOT  STUDY  ROOM  SETUP 
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APPENDIX  K.  WFTBN  PILOT  STUDY  SCHEDULE 


0800-0815 

Administration 

Informed  Consent 
Demographic  Questionnaire 

0815-0915 

Familiarity 

0915-0925 

Break 

0925  -  0930 

Familiarity  Test 

0930  -  0950 

Convoy  Express  101 

0950-  1005 

Knowledge  Test  1 

1005-  1015 

Break 

1015-1030 

Mission  Brief 

1030-  1045 

Mission  Planning 

1045-  1100 

Training  Wheels 

1100-1130 

Noble  Pass 

1130-  1230 

Lunch 

1230-  1245 

Debrief  Noble  Pass 

Post  Questionnaire  1 

1245-  1300 

Mission  Planning 

1300-  1400 

Rainbow  Canyon 

1400-  1430 

Reward  Exercise 

Catch  up 

Gays  Pass 

1430-  1450 

Knowledge  Test  2 

Post  Questionnaire  2 
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APPENDIX  L.  DEMOGRAPHIC  SURVEY  (EXPERIMENTS  2  -  3) 


Please  fill  in  the  following  questionnaire.  All  information  will  be  held 
confidential. 

1 .  Were  you  required  to  use  computer-based  simulations  as  part  of  your  training  in  the  past? 

NO /YES  (circle  one) 

If  YES,  what  simulations  have  you  used,  and  what  skills  were  they  used  to  train  (for  example, 
first-person  shooter,  flight  simulator,  operational  tactics,  etc.)? _ 


2.  Do  you  have  experience  playing  commercial  video  games? 

NO /YES  (circle  one) 

If  YES,  what  games  have  you  played,  and  what  kind  of  games  were  they  (for  example,  first- 
person  shooter,  flight  simulation,  sports,  fantasy,  etc.)? _ 


If  YES,  how  often  do  you  use  video  games? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

3.  Do  you  own  a  personal  computer? 

NO /YES  (circle  one) 

4.  How  often  do  you  use  a  computer  off  duty? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

5.  How  often  do  you  use  a  computer  on  duty? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

6.  What  hand  do  you  use  to  operate  a  computer  mouse? 

LEFT  /  RIGHT  /  EITHER  (circle  one) 

7.  How  often  do  you  use  the  following  computer  applications?  (please  answer  all  that  apply) 

a.  E-mail:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

b.  Browse  web:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

c.  Video:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

d.  Music:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

e.  Word:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

Processor 

f.  Other:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

8.  Year  of  birth:  _ 

9.  Primary  MOS  or  job  specialty:  _ 

10.  Rank:  _ Service:  _ 

1 1 .  Time  in  service:  _ years _ months  **  TURN  PAPER  OVER 
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12.  How  many  times  have  you  deployed  to  a  combat  theater? _ 

13.  If  you  deployed  to  a  combat  theater,  did  you  ever  participate  in  convoy  operations  as  a  driver,  assistant 
driver,  or  vehicle  gunner? 

NO  /  YES  (circle  one) 

If  YES,  approximately  how  many? _ 


14.  How  long  have  you  served  with  your  current  unit?  _ years _ months 

16.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for  each: 

DISAGREE  AGREE 


1 

2 

3 

4 

5 

Computer-based  simulation  is  an  effective  training  tool. 

Today’s  planned  computer-based  simulation  training  will 
improve  my  ability  to  conduct  conyoy  operations. 

Today’s  planned  computer-based  simulation  training  will 
improye  my  unit’s  ability  to  conduct  conyoy  operations. 

1  think  a  unit  should  use  computer-based  simulation  in  its 
tactical  training. 

17.  Write  the  number  of  times  you  have  used  a  sand  table  or  computer-based  simulation  for  each  of  the 
following  and  mark  your  preference: _ _ _ 


#  Sand  Table 

#  Simulation 

Preference  (circle  one) 

Training 

Sand  table 

/  Simulation 

Mission  Planning 

Sand  table 

/  Simulation 

Mission  Briefing 

Sand  table 

/  Simulation 

Mission  Rehearsal 

Sand  table 

/  Simulation 

18.  Rate  your  knowledge  of  the  following  small  unit  convoy  tactics,  techniques,  and  procedures  by  marking 
an  X  in  one  block  for  each  skill: 


NOT  AT  ALL  VERY 

PROFICIENT  PROFICIENT 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improyised  Explosiye  Deyice 

React  to  an  Improyised  Explosiye  Deyice  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  yehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuyer 

Shift  fires  /  cease  fires 

Vehicle  recoyery  /  bump  plan 

Casualty  eyacuation 

Communication  with  higher  headquarters 

Communication  between  yehicles  in  conyoy 

Communication  between  personnel  in  yehicle 
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APPENDIX  M.  DEMOGRAPHIC  SURVEY  (EXPERIMENT  4) 


Please  fill  in  the  following  questionnaire.  All  information  will  be  held  confidential. 

1 .  Were  you  required  to  use  computer-based  simulations  as  part  of  your  training  in  the  past? 

NO /YES  (circle  one) 

If  YES,  what  simulations  have  you  used,  and  what  skills  were  they  used  to  train  (for  example, 
first-person  shooter,  flight  simulator,  operational  tactics,  etc.)? _ 


2.  Do  you  have  experience  playing  commercial  video  games? 

NO /YES  (circle  one) 

If  YES,  what  games  have  you  played,  and  what  kind  of  games  were  they  (for  example,  first- 
person  shooter,  flight  simulation,  sports,  fantasy,  etc.)? _ 


If  YES,  how  often  do  you  use  video  games? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

3.  Do  you  own  a  personal  computer? 

NO /YES  (circle  one) 

4.  How  often  do  you  use  a  computer  off  duty? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

5.  How  often  do  you  use  a  computer  on  duty? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

6.  What  hand  do  you  use  to  operate  a  computer  mouse? 

LEFT  /  RIGHT  /  EITHER  (circle  one) 

7.  How  often  do  you  use  the  following  computer  applications?  (please  answer  all  that  apply) 

a.  E-mail:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

b.  Browse  web:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

c.  Video:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

d.  Music:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

e.  Word:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

Processor 

f.  Other:  Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

8.  Year  of  birth:  _ 

9.  Primary  MOS  or  job  specialty:  _ 

10.  Rank:  _ Service:  _ 

11.  Time  in  service:  _ years _ months  **  TURN  PAPER  OVER 
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12.  How  many  times  have  you  deployed  to  a  combat  theater? _ 

13.  If  you  deployed  to  a  combat  theater,  did  you  ever  participate  in  convoy  operations  as  a  driver, 
assistant  driver,  or  vehicle  gunner? 

NO /YES  (circle  one) 

If  YES,  approximately  how  many? _ _ 

14.  How  long  have  you  served  with  your  current  unit?  _ _years _ months 

16.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for  each: 


DISAGREE  AGREE 


1 

2 

3 

4 

5 

Computer-based  simulation  is  an  effective  training  tool. 

Today’s  planned  computer-based  simulation  training  will 
improve  imy  ability  to  conduct  convoy  operations. 

Today’s  planned  computer-based  simulation  training  will 
improve  mv  unit’s  ability  to  conduct  convoy  operations. 

1  think  a  unit  should  use  computer-based  simulation  in  its 
tactical  training. 

17.  Write  the  number  of  times  you  have  used  a  sand  table  or  computer-based  simulation  for 
each  of  the  following  and  mark  your  preference: _ _ 


#  Sand  Table 

#  Simulation 

Preference  (circle  one) 

Training 

Sand  table 

/  Simulation 

Mission  Planning 

Sand  table 

/  Simulation 

Mission  Briefing 

Sand  table 

/  Simulation 

Mission  Rehearsal 

Sand  table 

/  Simulation 

230 


18.  Rate  your  knowledge  of  the  following  small  unit  convoy  tactics,  techniques,  and  procedures 
by  marking  an  X  in  one  block  for  each  skill: 

NOT  AT  ALL  VERY 

PROFICIENT  PROFICIENT 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 

19.  Rate  your  unit’s  knowledge  of  the  following  small  unit  convoy  tactics,  techniques,  and 
procedures  by  marking  an  X  in  one  block  for  each  skill: 


NOT  AT  ALL  VERY 

EFFECTIVE  EFFECTIVE 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 
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APPENDIX  N.  POST-TRAINING  SURVEY  (EXPERIMENTS  2  -  4) 


1.  Circle  your  role  in  the  evaluation  exercise:  DRIVER  GUNNER  OTHER  (circle  one) 

If  you  were  not  a  driver  or  gunner,  what  was  your  role? _ 

2.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for  each: 


DISAGREE  AGREE 


1 

2 

3 

4 

5 

Computer-based  simulation  is  an  effective  training  tool. 

Today’s  computer-based  simulation  training  improved  imy 
ability  to  conduct  small  unit  tactical  convoy  operations. 

Today’s  computer-based  simulation  training  improved  imy 
unit’s  abilitv  to  conduct  small  unit  tactical  convov 
operations. 

I  think  a  unit  should  use  computer-based  simulation  in  its 
tactical  training. 

3.  If  you  had  to  choose  a  single  training  medium,  would  you  prefer  to  prepare  for  a  small  unit 
tactical  convoy  using  a  sand  table  or  a  computer-based  simulation? 

SAND  TABLE  /  COMPUTER  SIMULATION  (circle  one) 


4.  Rate  your  knowledge  of  the  following  small  unit  convoy  tactics,  techniques,  and  procedures  by 
marking  an  X  in  one  block  for  each: 


NOT  AT  ALL  VERY 

PROFICIENT  PROFICIENT 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 

233 


5.  Rate  the  effectiveness  of  this  computer  simulation  exercise  in  improving  your  unit’s 
performance  by  marking  an  X  in  one  block  for  each  skill: 


NOT  AT  ALL  VERY 

EFFECTIVE  EFFECTIVE 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 

6.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for  each: 

DISAGREE  AGREE 


1 

2 

3 

4 

5 

This  training  mission  was  successful. 

During  this  exercise,  I  felt  like  my  actions  in  the  virtual 
environment  had  no  consequences. 

During  this  exercise,  I  felt  like  I  was  playing  a  game. 

During  this  exercise,  I  felt  like  I  was  conducting  training. 

During  this  exercise,  I  felt  like  I  was  part  of  the  group 
working  together. 

During  this  exercise,  I  felt  isolated  from  the  others. 

This  computer  simulation  provided  sufficient  audio  cues 
for  me  to  know  what  was  going  on. 

This  computer  simulation  provided  sufficient  visual  cues 
for  me  to  know  what  was  going  on. 

The  training  value  of  this  exercise  came  from  the 
debriefing  and  not  the  exercise  itself. 
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7.  What  task  was  hardest  for  you? 


8.  What  task  was  easiest  for  you? 


9.  What  task  was  hardest  for  your  vehicle  crew? 


10.  What  task  was  easiest  for  your  vehicle  crew? 


1 1 .  What  task  was  hardest  for  your  unit? 


12.  What  task  was  easiest  for  your  unit? 


13.  How  did  this  simulation  exercise  help  you  or  your  unit? 


14.  How  did  this  simulation  exercise  waste  time  for  you  or  your  unit? 


15.  Do  you  have  any  suggestions  for  improving  the  convoy  training? 


16.  Provide  any  other  comments  or  suggestions  that  you  may  have. 

Commenti:  _ 


Commenti: 


Commenti: 


Commenti : 


Commenti: 


Thank  you  for  volunteering  to  participate  in  this  study. 
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APPENDIX  O.  POST-TRAINING  SURVEY  (EXPERIMENT  4 

CONTROL) 


1.  Circle  your  role  in  the  exercise:  DRIVER  GUNNER  VEHICLE  COMMANDER  (circle  one) 

If  you  were  a  unit  commander,  what  was  your  billet? _ 

2.  Rate  your  knowledge  of  the  following  small  unit  convoy  tactics,  techniques,  and  procedures  by  marking 
an  X  in  one  block  for  each: 


NOT  AT  ALL  VERY 

PROFICIENT  PROFICIENT 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 

3.  Rate  the  effectiveness  of  this  training  exercise  in  improving  your  unit’s  performance  by  marking  an  X  in 
one  block  for  each  skill: 


NOT  AT  ALL  VERY 

EFFECTIVE  EFFECTIVE 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 
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4.  What  task  was  hardest  for  you?  _ 

5.  What  task  was  easiest  for  you?  _ _ 

6.  What  task  was  hardest  for  your  vehicle  crew?  _ _ 

7.  What  task  was  easiest  for  your  vehicle  crew?  _ 

8.  What  task  was  hardest  for  your  unit?  _ 

9.  What  task  was  easiest  for  your  unit?  _ 

10.  How  did  this  training  exercise  help  you  or  your  unit? 

1 1 .  How  did  this  training  exercise  waste  time  for  you  or  your  unit? 

12.  Do  you  have  any  suggestions  for  improving  the  convoy  training? 

13.  Provide  any  other  comments  or  suggestions  that  you  may  have. 

Commenti:  _ 

Commenti:  _ 

Commenti:  _ 

Commenti:  _ 

Commenti:  _ 

Thank  you  for  volunteering  to  participate  in  this  study. 
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APPENDIX  P.  KNOWLEDGE  TEST  1  (EXPERIMENTS  3  -  4) 


1.  B  For  an  unblocked  ambush,  a  vehicle  in  the  kill  zone  should 

a.  Find  cover 

b.  Continue  to  move 

c.  Stop 

d.  Assault 

2.  B  Which  of  the  following  is  not  a  difference  between  short  and  long  security  halts? 

a.  Amount  of  dismounted  support  employed 

b.  Manning  crew  vehicle  crew  served  weapons 

c.  Amount  of  cover  employed  by  dismounts 

d.  Vulnerability  of  main  body 

3.  D  Which  action  sequence  best  describes  reaction  to  a  sniper? 

a.  Security  element  provides  support  by  fire  while  main  body  presses  through  the  kill 
zone 

b.  Security  element  provides  support  by  fire  while  main  body  assaults 

c.  Main  body  provides  support  by  fire  while  security  element  assaults 

d.  Vehicles  return  suppressive  fire,  speed  up,  and  keep  moving 

4.  C  Who  is  an  appropriate  person  to  conduct  lED  reporting? 

a.  Convoy  commander 

b.  Gunner  of  the  vehicle  who  spotted  the  lED 

c.  Assistant  convoy  commander 

d.  Security  element  commander 

5.  A  Which  part  of  the  convoy  typically  handles  navigation? 

a.  Head 

b.  Security 

c.  Combat  Assault 

d.  Main  body 

6.  A  When  analyzing  the  mission,  the  convoy  commander  should  follow  which  process? 

a.  Mission,  Enemy,  Terrain  and  weather.  Troops  and  fire  support.  Time  (METT-T) 

b.  Obstacles,  Key  terrain.  Observation  and  fields  of  fire.  Cover  and  concealment. 
Avenues  of  approach  (OKOCA) 

c.  Defend,  Reinforce,  Assault,  Withdraw,  Delay  (DRAW-D) 

d.  Confirm,  Clear,  Cordon,  Check,  Control  (5  C’s) 

7.  D  The  last  step  in  reacting  to  an  lED  is: 

a.  Cordon 

b.  Report  the  lED  to  higher  headquarters 

c.  Consolidate 

d.  Control 

8.  D  Where  in  the  convoy  should  the  convoy  commander  place  himself? 

a.  Lead  vehicle 

b.  Trail  vehicle 

c.  With  the  security  element  commander 

d.  Main  body 

9.  B  Which  is  one  of  the  three  elements  of  a  convoy  task  organization? 

a.  Trail 

b.  Security 

c.  Main  body 

d.  Head 
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10.  A  Which  is  the  first  step  in  the  recovery  of  a  disabled  vehicle? 

a.  Establish  local  security  at  the  vehicle  site 

b.  Report  to  higher  headquarters 

c.  Convoy  continues  past  the  vehicle 

d.  Nearest  vehicle  rigs  for  tow  with  strap,  chain,  cable,  or  tow  bar 


11.  B  In  order  to  maintain  flexibility  in  the  face  of  a  potential  threat,  a  convoy  commander 
should: 


a.  Focus  on  speed  and  momentum  throughout  the  convoy 

b.  Plan  an  alternate  route 

c.  Place  himself  in  the  lead  of  the  convoy 

d.  Place  himself  in  the  trail  of  the  convoy 


12.  C  Who  supervises  recovery  operations  for  the  convoy? 

a.  Vehicle  commander 

b.  Convoy  commander 

c.  Assistant  convoy  commander 

d.  Security  force  commander 


13.  D  Who  controls  the  employment  of  direct  and  indirect  fire  assets  for  the  convoy? 

a.  Vehicle  commander 

b.  Convoy  commander 

c.  Assistant  convoy  commander 

d.  Security  force  commander 


14.  C  Which  is  not  one  of  the  “5  C’s”? 

a.  Cordon 

b.  Control 

c.  Cover 

d.  Clear 


15.  A  When  a  convoy  reacts  to  an  unblocked  ambush,  the  most  critical  information  to  pass  to 
the  convoy  commander  is: 

a.  Direction  of  enemy  fires 

b.  Speed  of  the  lead  vehicle 

c.  Nearest  checkpoint  to  the  lead  vehicle 

d.  Number  of  vehicles  taking  fire 

16.  A  Which  is  an  appropriate  range  for  the  vehicle  interval  in  general  open  terrain? 

a.  50 -  100  meters 

b.  20 -50  meters 

c.  100  -  150  meters 

d.  150  -  200  meters 

17.  C  When  crossing  a  danger  area,  what  should  happen  after  the  trail  element  punches  up 
from  the  rear  to  provide  over  watch? 

a.  Convoy  resumes  normal  speed 

b.  Main  body  crosses  the  danger  area 

c.  Head  crosses  the  danger  area 

d.  Head  provides  over  watch  on  the  near  side 

18.  D  For  a  blocked  ambush,  a  vehicle  that  has  not  yet  entered  the  kill  zone  should 

a.  Assault 

b.  Continue  to  move 

c.  Speed  up 

d.  Find  cover 
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APPENDIX  Q.  KNOWLEDGE  TEST  2  (EXPERIMENTS  3  -  4) 


1 .  C  Which  of  the  following  should  not  be  done  at  a  short  security  halt? 

a.  Conduct  “5  and  25”  meter  checks 

b.  Maintain  360  degree  security 

c.  Dismounts  establish  a  secure  perimeter 

d.  Man  crew  served  weapons 

2.  B  Who  maintains  communication  with  higher  and  adjacent  authorities  for  the  convoy? 

a.  Vehicle  commander 

b.  Convoy  commander 

c.  Assistant  convoy  commander 

d.  Security  force  commander 

3.  C  Which  is  one  of  the  three  organizational  elements  of  a  convoy? 

a.  Security  element 

b.  Obstacle  Clearing  Detachment 

c.  Trail 

d.  Assault  force 

4.  B  When  crossing  a  danger  area,  which  element  provides  over  watch  while  the  main  body 
crosses? 

a.  Head 

b.  Trail 

c.  Main  body’s  security  elements 

d.  Combat  Assault  Element 

5.  A  Which  of  the  following  is  not  a  difference  in  reaction  between  blocked  and  unblocked 
ambushes? 

a.  Whether  escort  vehicles  maneuver  to  put  supporting  fires  on  the  enemy 

b.  Whether  vehicles  in  the  kill  zone  get  out 

c.  Whether  trail  elements  stop  and  seek  cover 

d.  Whether  an  alternate  route  is  considered 

6.  B  The  primary  mission  of  a  convoy  is: 

a.  Route  clearance 

b.  Moving  personnel  and  /  or  cargo 

c.  Intelligence  collection 

d.  Destroying  enemy  combatants 

7.  D  When  reacting  to  an  lED,  the  convoy  commander  should  follow  which  process? 

a.  Mission,  Enemy,  Terrain  and  weather.  Troops  and  fire  support.  Time  (METT-T) 

b.  Obstacles,  Key  terrain.  Observation  and  fields  of  fire.  Cover  and  concealment. 
Avenues  of  approach  (OKOCA) 

c.  Defend,  Reinforce,  Assault,  Withdraw,  Delay  (DRAW-D) 

d.  Confirm,  Clear,  Call,  Cordon,  Control  (5  C’s) 

8.  B  Which  of  the  following  should  not  be  used  to  determine  convoy  speed? 

a.  Terrain 

b.  Need  for  crew  rest 

c.  Weather  conditions 

d.  Likelihood  of  enemy  contact 

9.  A  Which  action  should  the  convoy  commander  avoid  when  reacting  to  a  sniper? 

a.  Stop 

b.  Speed  up 

c.  Return  fire 

d.  Use  smoke 
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10.  A  Higher  headquarters  maintains  situational  awareness  of  the  convoy  by  monitoring  the 
following  control  measure: 

a.  Checkpoints 

b.  Rally  points 

c.  Coordination  points 

d.  Waypoints 

11.  D  If  there  is  only  one  combat  life  saver,  where  in  the  convoy  should  he  be  placed? 

a.  With  the  convoy  commander 

b.  In  one  of  the  lead  vehicles 

c.  In  one  of  the  main  body  vehicles 

d.  In  one  of  the  trail  vehicles 

12.  C  Who  is  an  appropriate  person  to  conduct  casualty  reporting? 

a.  Combat  life  saver 

b.  Corpsman 

c.  Assistant  convoy  commander 

d.  Security  element  commander 

13.  D  When  a  convoy  vehicle  first  becomes  disabled,  the  security  element  commander  should 
consider: 

a.  Providing  local  security  at  the  vehicle  site 

b.  Punching  forward  to  determine  whether  enemy  are  waiting  to  ambush 

c.  Ensuring  the  convoy  stops  in  a  herringbone  in  case  of  air  attack 

d.  Providing  security  for  the  convoy  main  body 

14.  C  For  an  unblocked  ambush,  the  element  that  assaults  from  outside  the  kill  zone  is  the 

a.  Trail 

b.  Main  body 

c.  Security 

d.  Armored  escort 

15.  D  The  first  step  in  reacting  to  an  lED  is: 

e.  Cordon 

f.  Report  the  lED  to  higher  headquarters 

g.  Call  EOD 

h.  Confirm 

1 6.  D  Which  part  of  the  convoy  typically  sets  the  pace? 

a.  Head 

b.  Security 

c.  Combat  Assault 

d.  Main  body 

17.  A  Which  is  not  a  responsibility  of  the  vehicle  commander? 

a.  Observe  sectors  of  fire 

b.  Provide  direction  to  the  driver 

c.  Communicate  with  the  convoy  commander 

d.  Designate  sectors  of  observation 

18.  B  It  would  be  inappropriate  to  speed  up  for  which  immediate  action  drill? 

a.  Reaction  to  sniper 

b.  Reaction  to  unexploded  lED 

c.  Reaction  to  unblocked  ambush 

d.  All  of  the  above 
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APPENDIX  R.  OPORD  SHELL  (EXPERIMENT  4) 


Scenario: Combat  Convoy 


Common  Skill:  Convoy  Operations 
Unit  Level:  Platoon 

Condition:  The  platoon  has  received  a  fragmentary  order  (FRAGO)  to  conduct  a  combat 


Standard: 

patrol  through  the  designated  AO  in  order  to  deny  the  enemy  the  ability  to 
interfere  with  Coalition  vehicular  traffic  in  the  AO.  All  necessary  personnel  and 
equipment  are  available.  The  company  has  been  provided  guidance  on  the  rules 
of  engagement  (ROE)  and  /  or  rules  of  interaction  (ROI)  per  unit  SOP. 

The  platoon  conducts  in  accordance  with  tactical  standing  operating  procedures 
(TSOP),  the  order,  and  /  or  higher  commander's  guidance.  The  platoon  complies 
with  the  ROE  and  or  ROI. 
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Scenario  Mission  Order 


OPORD  #1 


Copy _ of _ copies 

XXXX 

Number- 1 

MOVEMENT  ORDER  (1)  (COMBAT  PATROL) 

BASIC  ORDER 

Ref:  (a)  Map:  Provided 

Time  Zone :  U 


X  Bn,  X  MARINES 
XXXXXXU  XXX 

Message  Ref 


Task  Organization:  No  change. 

I.  SITUATION 

A.  Terrain  and  Weather. 

29  PALMS:  Terrain  favors  the  defender.  The  terrain  provides  excellent 

observation  and  fields  of  fire  throughout  the  zone  allowing  defending 
forces  to  mass  fires  at  nearly  all  locations.  The  lack  of  cover  and 
concealment  at  most  points  enhances  this  capability.  Additionally, 
attacking  forces  are  required  to  move  through  severely  restrictive 
terrain  at  several  points  in  the  zone. 

SAHRANI :  Terrain  favors  the  defender.  The  restrictive  terrain 

canalizes  vehicle  movement,  while  high  ground  provides  excellent 
observation  and  fields  of  fire.  Wooded  areas  provide  good  concealment. 
The  terrain  is  suitable  for  small  unit  harassing  activity. 

1.  Obstacles.  Canalizing  mountain  ranges  per  map  provided. 

2.  Avenues  of  approach.  See  map  provided. 

3.  Key  terrain.  See  map  provided. 

4.  Observation  and  fields  of  fire.  See  map  provided. 

5.  Cover  and  concealment.  See  map  provided. 

6.  Weather.  Weather  favors  the  attacker.  With  clear  skies,  friendly 
forces  have  the  ability  to  observe  and  designate  targets  throughout  the 
zone,  thus  supporting  the  employment  of  artillery,  mortars,  and  fixed- 
and  rotary-wing 

close  air  support.  Prevailing  winds  favor  the  use  of  smoke  by  attacking 
forces.  There  is  little  to  no  precipitation  expected.  Humidity  is 
extremely 

low,  and  temperatures  vary  from  as  low  as  40  degrees  during  the  night  to 
as 

high  as  110  or  more  during  the  day. 
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B.  Enemy  Forces 

1.  Overview:  Combat  patrols  throughout  the  Coalition/ Joint  Operating 
Area  continue  to  be  targeted  by  insurgents.  Administrative  and 
logistical  constraints  continue  to  force  many  combat  patrols  to  travel 
along  predictable  routes  though  not  all  routes  are  experiencing  attacks. 
Virtually  all  combat  patrols  must  be  considered  possible  targets  for 
insurgents.  Smaller  combat  patrols  are  particularly  vulnerable.  As 
insurgent  experience  grows,  the  MARFOR  warns  of  increased  frequency, 
efficiency,  and  sophistication  of  attacks.  At  present,  roughly  5%  of  all 
combat  patrols  are  experiencing  some  type  of  attack  with  27  Coalition 
service  members  KIA  in  the  last  60  days.  Historically,  during  relief-in¬ 
place  operations,  attacks  have  increased  in  frequency  and  lethality. 

The  insurgents  continue  to  strike  primarily  at  night,  using  spotters 
with 

cell  phones  to  alert  attackers  of  approaching  combat  patrols.  However, 
in 

recent  weeks  due  to  insurgent  success  they  are  becoming  increasingly 
brazen 

with  more  attacks  occurring  during  daylight  hours.  The  pace  of  insurgent 
attacks  is  likely  to  increase  in  the  next  few  months  as  the  Coalition 
prepares  to  conduct  RIP  operations. 

2.  The  enemy  composition  has  been  sporadic  in  the  recent  past,  ranging 
from  single  shooters  to  platoon  sized  elements  executing  complex 
attacks,  integrating  direct  and  indirect  fires  with  victim  actuated, 
radio  controlled,  and  hard-wire  command  detonated  lEDs  and/or  obstacles. 
The  enemy  also  typically  uses  secondary  lEDs  in  conjunction  with  a 
primary  lED  attack  lOT  target  first  responders  and  security  forces.  The 
enemy  typically  uses  red  or  bright  pink  detonation  cord  to  prime  shots, 
and  white  or  copper  firing  wire  to  detonate  hard-wired  lEDs.  The  enemy 
has  indirect  capability  with  82mm  (range  3000m)  and  120mm  mortars  (range 
5700m);  the  mortars  have  been  used  primarily  for  harassing  fires  but 
have  been  effectively  integrated  into 

complex  attacks.  The  enemy  has  emplaced  multiple  AT  mines  along  the  MSRs 
and 

secondary  roads  but  does  not  have  the  engineering  capability  to  lay  a 
traditional  minefield.  The  mines  have  consisted  of  AT  mines;  the  AT 
mines 

have  been  used  in  a  double-stack  and  pressure  accumulation 
configurations  in 

a  100  m  (or  less)  stretch  of  road. 

3.  Enemy's  most  probable  course  of  action: 

a.  Observation:  Enemy  forces  have  the  capability  to  observe  coalition 
forces  upon  crossing  LD  and  will  be  able  to  provide  advance  warning  due 
to  their  spotters  using  long  range  cell  phones. 

b.  Indirect:  Enemy  fires  will  consist  of  harassing  fires  from  82mm  and 
120mm  mortars  with  possible  122mm  rocket  fire.  The  enemy  will  use  their 
indirect  fires  to  cover  their  emplaced  mines  and  obstacles  lOT  disrupt 
coalition  formations  and  to  prevent  CF  from  successfully  breaching 
obstacles . 
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c.  Direct:  Enemy  will  employ  direct  fire  with  a  variety  of  weapons 
including  AK47s  (max  eff  range-  200-300m),  RPKs  (800m),  FN  MAG  7.62 
(1200m)  and  RPG-7s  (moving  tgt-300m/stat  tgt-500m) .  Enemy  direct  fire 
engagements  will  range  from  single  shooters  to  integration  of  direct 
fires  with  IDF/IEDs/mines  and  obstacles  in  a  complex  attack/blocked 
ambush . 

d.  Obstacles:  Enemy  obstacles  will  most  likely  consist  of  deliberately 
emplaced  lEDs  made  with  varying  types  of  explosives  and  military 
ordnance.  The  enemy  has  been  known  to  initiate  lEDs  via  radio 
controlled,  hard  wired,  and  victim  actuated  methods.  Mines  are  also 
routinely  used  to  harass  CF  and  inflict  CF  casualties.  Two  to  four  mines 
are  typically  placed  on  the  road  and  shoulders  in  a  50  to  100  m  stretch 
of  road.  The  enemy  in  this  area  typically  uses  TM57  and/or  TM62  AT 
mines.  The  most  common  wire  used  for  hard  wired  lEDs  is  thin  copper  wire 
or  white  lamp  cord.  Detonation  cord  is  typically  red. 

e.  Chemical:  The  enemy  has  detonated  (1)  chlorine  enhanced  lED 
with  in  the  last  30  days. 

f.  Air:  Enemy  has  no  attack  aviation  capability. 

g.  Reserve:  The  enemy  has  little  or  no  capability  to  employ  a  reserve 
and  will  withdraw  to  another  ambush  site  or  attempt  to  blend  in  with  the 
civilian  populace. 

h.  Electronic:  The  enemy  has  limited  EW  capability;  they  have  the 
capability  of  eavesdropping  and  jamming  commercial  (i.e.  Motorola) 
radios  operating  on  unsecure  nets. 

4.  Enemy  most  dangerous  course  of  action:  The  enemy  will  conduct  a 
complex  attack  integrating  multiple  lEDs  or  mines  to  disrupt  the  CF 
combat  patrol,  employing  both  indirect  and  direct  fire  capabilities  to 
piecemeal  the  CF  patrol. 

C.  Friendly  Forces 

1.  Higher:  See  Annex  J 

2.  Adjacent:  See  Annex  J 

3.  Attachments  and  Detachments:  See  Annex  A 

4.  Legal  Considerations:  See  Annex  J 

II.  MISSION.  On  order,  X  Company,  Y  Battalion,  Z  Marines  conducts  combat 
patrols  through  the  designated  AO  to  destroy  enemy  forces  in  zone  in 
order  to  deny  the  enemy  the  ability  to  interfere  with  Coalition 
vehicular 

traffic  in  the  AO. 

III.  EXECUTION 

A.  Commander's  Intent  and  Concept  of  Operation 

1.  Commander's  Intent.  The  purpose  of  this  operation  is  to  gain 
contact  with  enemy  forces  and  destroy  them  in  order  to  create  conditions 
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for  the  enemy  that  make  their  efforts  to  interfere  with  Coalition 
operations  too  risky  to  be  worth  their  while.  The  method  we  will  use  is 
a  series  of  properly  coordinated,  "hardened"  combat  patrols  designed  to 
maximize  protection  to  troops  and  vehicles;  make  an  unattractive  target 
to  insurgents;  pose  as  a  deterrent  force  to  current  and  prospective 
insurgents;  and,  where  possible,  attrite  insurgent  forces.  The  desired 
end  state  is  significant  attrition  of  anti-coalition  forces  that  limits 
their  ability  to  recruit  new  fighters,  and  relative  freedom  of  movement 
along  key  MSRs  for  Coalition  forces. 

2.  Concept  of  the  operation:  See  Annex  C  (Operations) .  This  operation 
will  consist  of  three  phases.  Phase  I  is  the  assembly  area  phase  and 
consists  of  vehicle  and  personnel  preparation,  the  orders  process,  and 
pre-combat  checks  and  inspections.  Phase  II  is  the  movement  phase  and 
consists  of  the  actual  combat  patrol  movement  along  the  designated 
route.  Phase  III  is  the  reconstitution  phase.  Phase  III  is  on-order  and 
consists  of  Class  III  and  V  re-supply  and  follow-on  movement. 

B.  Tasks 

1 .  Patrol  Leader 

a.  At  DDTTTTU  MMM  YY,  conduct  a  combat  patrol  in  the  designated  AO 
lOT  deny  the  enemy  the  ability  to  disrupt  coalition  activities  in 
sector . 

b.  The  convoy  will  consist  of  no  more  than  8  firing  vehicles. 

c.  Task  the  Assistant  Patrol  Leader  and  subordinate  element  leaders 
as  necessary. 

d.  Coordinate  movement  during  Phase  II  with  X  Battalion,  X  Marines 
COC. 

e.  Designate  command,  control,  and  communications  assignments  and 
techniques  that  will  be  employed  to  carry  out  the  move. 

f.  Properly  integrate  attachments. 

g.  Execute  necessary  orders,  rehearsals,  pre-combat  checks,  and 
inspections . 

h.  Complete  Convoy  After  Action  Report  per  SOP. 

2.  Assistant  Patrol  Leader 

a.  At  DDTTTTU  MMM  YY,  assist  in  the  conduct  of  a  combat  patrol  in  the 
designated  AO  lOT  deny  the  enemy  the  ability  to  disrupt  coalition 
activities  in  sector. 

b.  Supervise  marshalling  activities  in  the  Assembly  Area. 

c.  Supervise  rehearsals,  pre-combat  checks,  and  inspections. 

d.  Supervise  SOP  reiteration  and  enforcement. 

e.  Supervise  enroute  maintenance  and  accountability. 
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C.  Coordinating  Instructions:  See  Annex  C  Appendix  18  (Operations 
Overlay) 

1.  Rank  X.  X.  XXXXXX  is  designated  convoy  commander. 

2.  Designated  elements  OPCON  to  convoy  commander  NLT  24  hours  prior  to 
execution . 

3.  Convoy  task  organization  will  terminate  on  order. 

4.  MOPP  Level  0  in  effect. 

5.  COMBAT  PATROL  ASSEMBLY  AREA:  designated  per  map  provided 

IV.  ADMINISTRATION  &  LOGISTICS 

A.  Administration.  Additional  Control  Measures:  Used  to  organize  and 
report  combat  patrol  movement  and  position: 

CP  GRID  LOCAL  DESIGNATION 

See  scenario  map. 

B.  Logistics 

1.  MREs  issued  as  required. 

2.  Ammunition  staged  in  Assembly  Area;  issued  per  combat  patrol 
Commander's  guidance. 

3.  CASEVAC  support  provided  by  DET-XX. 

4.  Patrols  and  convoys  will  tow  all  damaged  vehicles  to  the  release 
point . 

V.  COMMAND  AND  SIGNAL 

A.  Command  Relationships.  Per  Battalion  SOP. 

B.  Command  Posts  and  Headquarters.  Located  at  Camp  Wilson. 

C.  Signal.  Per  SOP. 


ACKNOWLEDGE  RECEIPT 


R.  F.  DAGGER 
COL,  USMC 
COMMANDING 

ANNEXES : 

None . 

OFFICIAL: 

s/ 

C.  J.  SABER 
LTCOL,  USMC 

OPERATIONS  OFFICER 
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APPENDIX  S.  LIVE  CONVOY  EVALUATION  (EXPERIMENT  4) 


MOTORIZED  PATROL  CHECKLIST 

CO. _ PEL _ SEC _ DATE _ TIME _ 

PATROL  LEADER- _ #VICS/PAX _ 

1,  Prior  to  DFL 

□  WamO  issued? _ 

□  FragO  issued? _ 

□  Final  coordination  and  intelligence  brief _ 

□  PCC/PCCI _ 

4  5  6  7  8  9  10 

□  Mission  card  submitted _ 

□  _ 

2.  Depart  Friendly  Lines 

□  Short  count _ 

□  Comm  check,  move  to  AA _ 

l|  2|  sj  4|  5|  6|  ?!  8|  9|  10  I 

□  Dismount  &  Condition  4 _ 

□  Check  out  w/  COC _ 

□  _ 

3.  Patrol  Movement 

□  Navigator  designated/employed? _ 

□  PL  positioned  for  best  C2? _ 

□  Lead  trace  reporting  for  turns  or  direction  changes _ 

□  Use  of  Vehicle  Control  Measures _ 

l|  2|  3|  4|  5|  6|  ?!  8|  9|  10  | 

□  Dispersion  and  speed  appropriate  to  METT _ 

l|  2^  3|  4|  5^  6|  ?!  8|  9|  10  | 

□  Use  of  checkpoints/HHQ  reporting _ 

l|  I  2|  I  3^  4|  I  5^  6|  I  ?!  I  8|  I  9|  |  10  | 

□  _ 

4,  Immediate  Actions 

a.  Short  Security  Halt 

□  Duration  per  SOP? _ 

□  Fire/observation  sectors  covered? _ 

l|  2|  3|  4|  5|  6|  l\  8|  9|  10 | 

□ 
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b. 

Lons  Security  Halt 

□ 

Position  improvement 

1 

□ 

4  5 

C 

]3E 

]C 

J  10 

□ 

Security  established 

1 

□ 

4  5 

]C 

□ 

o 

1 — 1 

□ 

c. 

PIED  spotted 

□ 

Brevity  code  used? 

□ 

Swee 

p/search  techniques  viable?  _ 

1 

□ 

3 

1  3U 

6 

] 

]  c 

]  c 

]  10 

□ 

d. 

lED  found 

□ 

Confirm 

1 

□ 

2 

3 

4  5 

C 

] 

]  c 

J  10 

□ 

Cordon 

1 

□ 

2 

3 

4  5 

c 

] 

]  c 

J  10 

□ 

Control 

1 

□ 

1  ^  ^ 

c 

] 

]  c 

J  10 

□ 

Call_ 

□ 

Clear 

□ 

e. 

Vehicle  Recovery 

□ 

Rehearsal  evident? 

1 

□ 

2 

3 

4  5 

c 

] 

]  c 

□ 

o 

□ 

Recovery  vel 

licle  designated 

□ 

Equipment  available 

□ 

Obscuration/deception  used? 

□ 
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/  Casualty  Evacuation 

□  Rehearsal  evident? _ 

1 1  I  2 1  I  3 1  I  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

□  Aid  and  litter  teams  or  vehieles _ 

□  HHQ  reporting _ 

□  Triage  and  mission  impact _ 

1 1  I  2 1  I  3 1  I  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

□  Handling  of  civilians _ 

1 1  I  2 1  3 1  I  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 


g.  Actions  on  Contact 

□  PID  of  targets _ 

l|  I  2 1  I  3rn  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

□  Sectors  of  fire _ 

l|  I  2 1  I  3 1  I  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

□  Fire  discipline _ 

l|  I  2 1  3 1  I  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

□  Rehearsed  response? 

l|  I  2 1  I  3 1  I  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

□  Individual  actions  support  PL  SQM 

l|  I  2 1  I  3 1  I  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

□  Vehicle-dismount  integration _ 

l|  I  2 1  I  3 1  I  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

□  CSW  fire  support  shifted  and/or  ceased 

l|  I  2 1  I  3 1  I  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

□  Consolidation  on  objective _ 

l|  I  2 1  I  3 1  I  4 1  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

□  Pursuit  and  continuing  actions _ 

l|  I  2 1  I  3 1  I  4 1  I  5 1  I  6 1  I  7 1  I  8 1  |  9|  |  lo| 

1 .  Reporting  to  HHQ 

2.  Casualties 

3.  Detainees 

4.  Impact  on  civil  populace 

5.  Accountability 

6.  Redistribution/recovery 
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5,  Vehicle  Control  Points 

a.  Snap  VCP 


□  Proper  personnel  seareh _ 

□  Use  of  interpreter  or  HN  forees 

□ 


b. 

□ 

□ 

□ 

□ 


□ 

□ 

□ 


□ 

□ 

□ 

□ 

□ 

□ 

□ 


Hasty  VCP 

Long  halt  seeurity _ 

Rehearsal  evident? _ 

Barrier  plan  effeetive 


HHQ  reporting _ 

Personnel  search  techniques _ 

Vehicle  search  techniques _ 

Intelligence  collection _ 

Use  of  interpreter  or  HN  forces 
Basic  Arabic  commands  used? 


c.  Deliberate  VCP 

i.  Not  employed 


6,  Re-entering  Friendly  Lines 

□  Request  made  before  reaching  gate  _ 

□  VIC/PAX  count  modified,  if  needed 

□  Dismount  and  go  Condition  4 _ 

□  Conduct  AAR _ 

□  Detainee  procedures _ 

□  Prepare  for  next  patrol _ 

Additional  Comments: 
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APPENDIX  T.  STUDENT  DEMOGRAPHIC  SURVEY 

(EXPERIMENT  5) 


Please  fill  in  the  following  questionnaire.  All  information  will  be  held  confidential. 

1 .  Were  you  required  to  use  computer-based  simulations  as  part  of  your  training  in  the  past? 

NO  /  YES  (circle  one) 

If  YES,  what  simulations  have  you  used,  and  what  skills  were  they  used  to  train  (for  example,  first- 
person  shooter,  flight  simulator,  operational  tactics,  etc.)? _ 


2.  Do  you  have  experience  playing  commercial  video  games? 

NO  /  YES  (circle  one) 

If  YES,  what  games  have  you  played,  and  what  kind  of  games  were  they  (for  example,  first-person 
shooter,  flight  simulation,  sports,  fantasy,  etc.)? _ 


If  YES,  how  often  do  you  use  video  games? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

3.  Do  you  own  a  personal  computer? 

NO  /  YES  (circle  one) 

4.  How  often  do  you  use  a  computer  off  duty? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

5.  How  often  do  you  use  a  computer  on  duty? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

6.  What  hand  do  you  use  to  operate  a  computer  mouse? 

LEFT  /  RIGHT  /  EITHER  (circle  one) 


7.  How  often  do  you  use  the  following  computer  applications?  (please  answer  all  that  apply) 


a.  E-mail:  Approximately 

b.  Browse  web:  Approximately 


c.  Video: 

d.  Music: 

e.  Word: 
Processor 

f.  Other: 

8.  Year  of  birth: 


Approximately 

Approximately 

Approximately 

Approximately 


9.  Primary  MOS  or  job  specialty: 

10.  Rank: 


hours  per  day  /  week  /  month  /  year  (circle  one) 
hours  per  day  /  week  /  month  /  year  (circle  one) 
hours  per  day  /  week  /  month  /  year  (circle  one) 
hours  per  day  /  week  /  month  /  year  (circle  one) 
hours  per  day  /  week  /  month  /  year  (circle  one) 

hours  per  day  /  week  /  month  /  year  (circle  one) 


1 1 .  Time  in  service: 


years 


_ Service: 

months 


TURN  PAPER  OVER 


253 


12.  How  many  times  have  you  deployed  to  a  combat  theater? _ 

13.  If  you  deployed  to  a  combat  theater,  did  you  ever  participate  in  convoy  operations  as  a  driver, 
assistant  driver,  or  vehicle  gunner? 

NO /YES  (circle  one) 

If  YES,  approximately  how  many? _ _ 

14.  How  long  have  you  served  with  your  current  unit?  _ _years _ months 

16.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for  each: 


DISAGREE  AGREE 


1 

2 

3 

4 

5 

Computer-based  simulation  is  an  effective  training  tool. 

Today’s  planned  computer-based  simulation  training  will 
improve  imy  ability  to  conduct  convoy  operations. 

Today’s  planned  computer-based  simulation  training  will 
improve  mv  unit’s  ability  to  conduct  convoy  operations. 

1  think  a  unit  should  use  computer-based  simulation  in  its 
tactical  training. 

17.  Write  the  number  of  times  you  have  used  a  sand  table  or  computer-based  simulation  for 
each  of  the  following  and  mark  your  preference: _ _ 


#  Sand  Table 

#  Simulation 

Preference  (circle  one) 

Training 

Sand  table 

/  Simulation 

Mission  Planning 

Sand  table 

/  Simulation 

Mission  Briefing 

Sand  table 

/  Simulation 

Mission  Rehearsal 

Sand  table 

/  Simulation 
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18.  Rate  your  knowledge  of  the  following  small  unit  convoy  tactics,  techniques,  and  procedures  by  marking  an  X  in 
one  block  for  each  skill: 


NOT  AT  ALL  VERY 

PROFICIENT  PROFICIENT 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 

Execute  the  troop  leading  steps  using  BAMCIS 

Conduct  mission  analysis  using  METT-T 

Receive  a  5  paragraph  operational  order 

Give  a  5  paragraph  operational  order 

19.  Rate  your  unit’s  knowledge  of  the  following  small  unit  convoy  tactics,  techniques,  and  procedures  by  marking 
an  X  in  one  block  for  each  skill: 


NOT  AT  ALL  VERY 

EFFECTIVE  EFFECTIVE 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 
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APPENDIX  U.  INSTRUCTOR  DEMOGRAPHIC  SURVEY 

(EXPERIMENT  5) 


Please  fill  in  the  following  questionnaire.  All  information  will  be  held  confidential. 

1 .  Were  you  required  to  use  computer-based  simulations  as  part  of  your  training  in  the  past? 

NO  /  YES  (circle  one) 

If  YES,  what  simulations  have  you  used,  and  what  skills  were  they  used  to  train  (for  example,  first- 
person  shooter,  flight  simulator,  operational  tactics,  etc.)? _ 


2.  Do  you  have  experience  playing  commercial  video  games? 

NO  /  YES  (circle  one) 

If  YES,  what  games  have  you  played,  and  what  kind  of  games  were  they  (for  example,  first-person 
shooter,  flight  simulation,  sports,  fantasy,  etc.)? _ 


If  YES,  how  often  do  you  use  video  games? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

3.  Do  you  own  a  personal  computer? 

NO  /  YES  (circle  one) 

4.  How  often  do  you  use  a  computer  off  duty? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

5.  How  often  do  you  use  a  computer  on  duty? 

Approximately _ hours  per  day  /  week  /  month  /  year  (circle  one) 

6.  What  hand  do  you  use  to  operate  a  computer  mouse? 

LEFT  /  RIGHT  /  EITHER  (circle  one) 


7.  How  often  do  you  use  the  following  computer  applications?  (please  answer  all  that  apply) 


a.  E-mail:  Approximately 

b.  Browse  web:  Approximately 


c.  Video: 

d.  Music: 

e.  Word: 
Processor 

f.  Other: 

8.  Year  of  birth: 


Approximately 

Approximately 

Approximately 

Approximately 


9.  Primary  MOS  or  job  specialty: 

10.  Rank: 


hours  per  day  /  week  /  month  /  year  (circle  one) 
hours  per  day  /  week  /  month  /  year  (circle  one) 
hours  per  day  /  week  /  month  /  year  (circle  one) 
hours  per  day  /  week  /  month  /  year  (circle  one) 
hours  per  day  /  week  /  month  /  year  (circle  one) 

hours  per  day  /  week  /  month  /  year  (circle  one) 


1 1 .  Time  in  service: 


years 


_ Service: 

months 


TURN  PAPER  OVER 
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12.  How  many  times  have  you  deployed  to  a  combat  theater? _ 

13.  If  you  deployed  to  a  combat  theater,  did  you  ever  participate  in  convoy  operations  as  a  driver, 
assistant  driver,  or  vehicle  gunner? 

NO /YES  (circle  one) 

If  YES,  approximately  how  many? _ _ 

14.  How  long  have  you  served  with  your  current  unit?  _ _years _ months 

16.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for  each: 


DISAGREE  AGREE 


1 

2 

3 

4 

5 

Computer-based  simulation  is  an  effective  training  tool. 

Computer-based  simulation  training  will  enable  me  to 
train  my  Marines  better  than  if  1  did  not  have  it. 

1  think  a  unit  should  use  computer-based  simulation  in  its 
tactical  training. 

17.  Write  the  number  of  times  you  have  used  a  sand  table  or  computer-based  simulation  for 
each  of  the  following  and  mark  your  preference: _ _ 


#  Sand  Table 

#  Simulation 

Preference  (circle  one) 

Training 

Sand  table 

/  Simulation 

Mission  Planning 

Sand  table 

/  Simulation 

Mission  Briefing 

Sand  table 

/  Simulation 

Mission  Rehearsal 

Sand  table 

/  Simulation 
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18.  Rate  how  effective  you  anticipate  VBS  2™  to  be  in  training  each  of  the  following  small  unit  convoy 
tactics,  techniques,  and  procedures  by  marking  an  X  in  one  block  for  each  skill: 

NOT  AT  ALL  VERY 

EFFECTIVE  EFFECTIVE 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 

Execute  the  troop  leading  steps  using  BAMCIS 

Conduct  mission  analysis  using  METT-T 

Receive  a  5  paragraph  operational  order 

Give  a  5  paragraph  operational  order 

19.  Rate  your  proficiency  for  each  of  the  following  trainer  skills  with  VBS  2™; 

NOT  AT  ALL  VERY 

PROFICIENT  PROFICICENT 


Set  up  the  network  and  computer  hardware 

1 

2 

3 

4 

5 

Set  up  VBS  2^“  for  Administrator  and  User 

Start  a  scenario  that  has  already  been  built 

Add  entities  to  a  scenario 

Use  triggers 

Control  civilians  and  traffic 

Control  enemy 

Maintain  your  situational  awareness  as  the  trainer 

Set  up  CNR  Sim  and  CNR  Log 

Use  an  AAR  to  facilitate  a  debrief 

Answer  user’s  questions  about  how  to  use  VBS  2™ 

Facilitate  a  call  for  fire 

Use  the  simulation  to  meet  your  training  objectives 
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20.  As  a  trainer,  what  training  objective  or  skill  do  you  think  VBS  2™  will  help  you  the  most  to 
train  your  Marines? _ 


21.  As  a  trainer,  what  training  objective  or  skill  do  you  think  VBS  2™  will  help  you  the  least  to 
train  your  Marines? _ 


22.  How  would  you  improve  this  train-the-trainer  instructional  package? 
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APPENDIX  V.  INSTRUCTOR  TRAINING  NOTES  (EXPERIMENT  5) 


❖  Setting  up  the  DVTE  suite’s  hardware 

>  Setting  up  computers 

>  Connecting  the  LAN 

❖  Disseminate  appropriate  files  to  all  user  computers 

>  Administrator 

■  Go  to  Windows  Explorer 

■  Click  on  appropriate  file 

■  Go  to  File-Properties  and  then  the  “Sharing”  tab 

■  Under  “Network  sharing  and  security”  check  the  “Share  this  folder  on  the  network” 
box 

>  User 

■  Go  to  Tools-Map  Network  Drive,  click  browse 

■  The  shared  folder  should  show  up;  click  on  it,  and  click  finish 

❖  Learning  to  be  a  VBS  2  user 

>  Introduction  brief 

>  Cheat  Sheet 

>  Familiarity  scenario 

>  Training  scenarios 

>  Library 
CNR 

>  Documentation  and  access  through  All  Programs^Bohemia  Interactive^CNR 

>  Setting  up  CNR  Log 

■  Configuration-Remote  Access^Check  “Remote  Access  Enabled” 

■  Ensure  Access  Port  is  8080 

■  Leave  open 

>  SetupVBSLVC 

■  Open  Notepad 

■  Open  c:\Program  Files\Bohemia  lnteractive\VBS2  VTK\config\VBSCIient.config 

■  Get  computer  IP  address 

•  Control  Panel-Network  Connections-Local  Area  Connection-Support  tab 

■  Change  IP  address  in  VBSClient.config  to  server  computer 

>  Start  CNR  Sim  on  client  computers 

■  Set  up  channels,  frequencies,  and  hot  keys 

>  Start  server  computer  VBS  2  using  batch  file  in  source  directory 

■  c:\Program  Files\Bohemia  lnteractive\VBS2  VTK\VBS2  LVC  Game  CNR  Log. bat 

>  Start  client  computers  in  normal  VBS  2 

>  CNR  Log  must  be  on  for  AAR  function  to  work 

❖  Show  location  of  manuals 

>  Go  to  c:\Program  Files\Bohemia  lnteractive\VBS2  VTK\docs 

❖  Run  Training  Wheels 

>  Use  of  overhead  view 

❖  Setting  up  the  options 

>  File  is  My  Documents\VBS2\AdministratorVBS2PROFILE  file 

>  Video  Options 

■  Terrain  Distance:  change  to  10,000 

■  Resolution:  1280x  1024x32 

■  Advanced  tab:  Object  Distance:  10,000 

>  Controls 

■  Up:  Delete  “Page  Up” 

■  Go  Prone:  Delete  “Page  Down” 

■  Map  and  Hide  Map:  Delete  “M”  and  replace  with  “Right  CTRL  M” 
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■  GPS  and  GPS  Toggle:  “M”  and  “2  x  M” 

■  Voice  over  net  and  Push  to  talk:  Delete  “Caps  Lock” 

■  Talk  on  Vehicle  channel:  “Caps  Lock” 

Building  a  scenario 

>  Entering  as  Administrator  (Administrator,  DVTEM90build2) 

>  “File”  options 

>  “View”  options 

>  Map  manipulation:  zoom  and  slide 

>  Adding  a  unit 

■  Add  a  server,  civilian  or  blue 

■  Naming,  Rank,  Make  Player 

■  Review  your  work 

■  Add  an  enemy  player 

■  Moving  and  turning  units 

■  Deleting  a  unit 

>  Add  a  group 

■  Add  an  enemy  fire  team 

>  Add  a  vehicle 

■  Unmanned  Vehicles — MQ-1  Predator  (USAF) 

>  Add  an  empty  vehicle 

■  Give  yourself  a  HMMWV 

>  Measure  distance  tool 

■  Add  a  measure  distance  point,  right  click,  new  distance 

>  Add  a  waypoint 

■  Add  a  waypoint,  and  then  add  more  to  make  a  circular  path 

■  For  the  last  waypoint  in  the  circle,  cross 

•  Edit  the  waypoint  and  change  “MOVE”  to  “CYCLE” 

>  Add  a  vehicle^Ground  control  station 

>  Add  a  control  link 

■  Right  click  and  “Link  to  vehicle”  and  then  click  on  Ground  control  station  to  link 

■  Player  walks  up  to  the  ground  control  station  and  treats  like  vehicle  (“U”  to  interact, 
then  scroll  to  UAV  controller) 

>  Add  a  marker  (useful  for  2D  view) 

>  Intel:  change  the  weather  and  time 

>  Add  an  I  ED 

>  Add  a  trigger 

■  lED 

■  Artillery  strike 

■  Unit  /  vehicle  movement 

Use  of  AAR  tool  to  debrief  an  exercise  scenario 
Serving  as  controller  for  an  exercise  scenario 

>  Artillery 

>  Civilians 

>  T  raffic 

>  Use  of  triggers  to  cause  casualties  and  vehicle  breakdown 
Use  of  indirect  fire  console 

>  Visibility  settings 

>  Putting  the  console  in  a  mission 

>  Les 

>  Running  the  console 

■  Different  mission  types 

■  Initial  call  for  fire 

■  Adjustments 

■  Fire  for  effect 

>  Using  the  console  in  a  mission 
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APPENDIX  W.  STUDENT  POST  TRAINING  SURVEY 

(EXPERIMENT  5) 


1.  Circle  your  role  in  the  exercise:  DRIVER  GUNNER  VEHICLE  COMMANDER  (circle  one) 

If  you  were  a  unit  commander,  what  was  your  billet? _ 

2.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for  each: 


DISAGREE  AGREE 


1 

2 

3 

4 

5 

Computer-based  simulation  is  an  effective  training  tool. 

Today’s  computer-based  simulation  training  improved  my  ability 
to  conduct  small  unit  tactical  convoy  operations. 

Todav’s  comouter-based  simulation  trainino  improved  mv  unit’s 
ability  to  conduct  small  unit  tactical  convoy  operations. 

I  think  a  unit  should  use  computer-based  simulation  in  its 
tactical  training. 

3.  If  you  had  to  choose  a  single  training  medium,  would  you  prefer  to  prepare  for  a  small  unit  tactical  convoy 
using  a  sand  table  or  a  computer-based  simulation? 

SAND  TABLE  /  COMPUTER  SIMULATION  (circle  one) 

4.  Rate  your  knowledge  of  the  following  small  unit  convoy  tactics,  techniques,  and  procedures  by  marking 
an  X  in  one  block  for  each: 


NOT  AT  ALL  VERY 

PROFICIENT  PROFICIENT 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 

Execute  the  troop  leading  steps  using  BAMCIS 

Conduct  mission  analysis  using  METT-T 

Receive  a  5  paragraph  operational  order 

Give  a  5  paragraph  operational  order 
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5.  Rate  the  effectiveness  of  this  computer  simulation  exercise  in  improving  your  unit’s 
performance  by  marking  an  X  in  one  block  for  each  skill: 


NOT  AT  ALL  VERY 

EFFECTIVE  EFFECTIVE 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 

6.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for  each: 

DISAGREE  AGREE 


1 

2 

3 

4 

5 

This  training  mission  was  successful. 

During  this  exercise,  I  felt  like  my  actions  in  the  virtual 
environment  had  no  consequences. 

During  this  exercise,  I  felt  like  I  was  playing  a  game. 

During  this  exercise,  I  felt  like  I  was  conducting  training. 

During  this  exercise,  I  felt  like  I  was  part  of  the  group 
working  together. 

During  this  exercise,  I  felt  isolated  from  the  others. 

This  computer  simulation  provided  sufficient  audio  cues 
for  me  to  know  what  was  going  on. 

This  computer  simulation  provided  sufficient  visual  cues 
for  me  to  know  what  was  going  on. 

The  training  value  of  this  exercise  came  from  the 
debriefing  and  not  the  exercise  itself. 
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7.  What  task  was  hardest  for  you? 


8.  What  task  was  easiest  for  you? 


9.  What  task  was  hardest  for  your  vehicle  crew? 


10.  What  task  was  easiest  for  your  vehicle  crew? 


1 1 .  What  task  was  hardest  for  your  unit? 


12.  What  task  was  easiest  for  your  unit? 


13.  How  did  this  simulation  exercise  help  you  or  your  unit? 


14.  How  did  this  simulation  exercise  waste  time  for  you  or  your  unit? 


15.  Do  you  have  any  suggestions  for  improving  the  convoy  training? 


16.  Provide  any  other  comments  or  suggestions  that  you  may  have. 

Commenti:  _ 


Commenti: 


Commenti: 


Commenti: 


Commenti: 


Thank  you  for  volunteering  to  participate  in  this  study. 
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APPENDIX  X.  INSTRUCTOR  POST  TRAINING  SURVEY 

(EXPERIMENT  5) 


1.  Circle  your  role  in  the  exercise:  CONTROLLER  ARTILLERY  OBSERVER  (circle  one) 

2.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for  each: 


DISAGREE  AGREE 


1 

2 

3 

4 

5 

Computer-based  simulation  is  an  effective  training  tool. 

Computer-based  simulation  training  enabled  me  to  train  my 
Marines  better  than  if  I  did  not  have  it. 

I  think  a  unit  should  use  computer-based  simulation  in  its 
tactical  training. 

3.  If  you  had  to  choose  a  single  training  medium,  would  you  prefer  to  prepare  for  a  small  unit  tactical 
convoy  using  a  sand  table  or  a  computer-based  simulation? 

SAND  TABLE  /  COMPUTER  SIMULATION  (circle  one) 

4.  Rate  how  effective  you  thought  VBS  2™  was  in  training  each  of  the  following  small  unit  convoy 
tactics,  techniques,  and  procedures  by  marking  an  X  in  one  block  for  each  skill: 


NOT  AT  ALL  VERY 

EFFECTIVE  EFFECTIVE 


1 

2 

3 

4 

5 

React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 

Execute  the  troop  leading  steps  using  BAMCIS 

Conduct  mission  analysis  using  METT-T 

Receive  a  5  paragraph  operational  order 

Give  a  5  paragraph  operational  order 
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5.  Rate  which  training  platform  you  would  prefer  to  use  to  train  each  of  the  following  small  unit 
convoy  tactics,  techniques,  and  procedures  by  marking  an  X  in  one  block  for  each  skill: 


VBS  2™  Don’t  care  VCCT 


React  to  an  unexploded  Improvised  Explosive  Device 

React  to  an  Improvised  Explosive  Device  detonation 

Take  immediate  action  against  a  blocked  ambush 

Take  immediate  action  against  an  unblocked  ambush 

Cordon  and  360  degree  security 

Employ  vehicle  machine  guns  /  weapons 

Mounted  fire  and  maneuver 

Shift  fires  /  cease  fires 

Vehicle  recovery  /  bump  plan 

Casualty  evacuation 

Communication  with  higher  headquarters 

Communication  between  vehicles  in  convoy 

Communication  between  personnel  in  vehicle 

Execute  the  troop  leading  steps  using  BAMCIS 

Conduct  mission  analysis  using  METT-T 

Receive  a  5  paragraph  operational  order 

Give  a  5  paragraph  operational  order 
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6.  Rate  your  proficiency  for  each  of  the  following  trainer  skills  with  VBS  2™: 


NOT  AT  ALL  VERY 

PROFICIENT  PROFICIENT 


1 

2 

3 

4 

5 

Set  up  the  network  and  computer  hardware 

Set  up  VBS  2™  for  Administrator  and  User 

Start  a  scenario  that  has  already  been  built 

Add  entities  to  a  scenario 

Use  triggers 

Control  civilians  and  traffic 

Control  enemy 

Maintain  your  situational  awareness  as  the  trainer 

Set  up  CNR  Sim  and  CNR  Log 

Use  an  AAR  to  facilitate  a  debrief 

Answer  user’s  questions  about  how  to  use  VBS  2™ 

Facilitate  a  call  for  fire 

Use  the  simulation  to  meet  your  training  objectives 

7.  Rate  your  agreement  with  the  following  statements  by  marking  an  X  in  one  block  for  each: 

DISAGREE  AGREE 


1 

2 

3 

4 

5 

This  training  was  successful. 

During  this  exercise,  I  felt  like  training  was  not  real 
enough. 

During  this  exercise,  I  felt  like  we  were  playing  a  game. 

During  this  exercise,  I  felt  like  we  were  conducting 
training. 

This  computer  simulation  provided  sufficient  audio  cues 
for  me  to  know  what  was  going  on. 

This  computer  simulation  provided  sufficient  visual  cues 
for  me  to  know  what  was  going  on. 

The  training  value  of  this  exercise  came  from  the 
debriefing  and  not  the  exercise  itself. 
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8.  As  a  trainer,  what  training  objective  or  skill  do  you  think  VBS  2™  will  help  you  the  most  to  train 
your  Marines? _ 


9.  As  a  trainer,  what  training  objective  or  skill  do  you  think  VBS  2™  will  help  you  the  least  to  train 
your  Marines? _ 


10.  How  did  this  simulation  exercise  help  you  or  your  unit? 


1 1 .  How  did  this  simulation  exercise  waste  time  for  you  or  your  unit? 


12.  Do  you  have  any  suggestions  for  improving  the  simulation  training? 


13.  Provide  any  other  comments  or  suggestions  that  you  may  have. 

Comment:  _ 


Comment: 


Comment: 


Comment: 


Comment: 


Thank  you  for  volunteering  to  participate  in  this  study. 
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APPENDIX  Y.  KNOWLEDGE  TEST  A  (EXPERIMENT  5) 


1.  B  For  an  unblocked  ambush,  a  vehicle  in  the  kill  zone  should 

e.  Find  cover 

f.  Continue  to  move 

g.  Stop 

h.  Assault 


lA  Procedure 

Unblocked 
ambush — does 
student 

understand  basic 
execution? 


2.  A 


3.  C 


Higher  headquarters  maintains  situational  awareness  of  the  convoy 
by  monitoring  the  following  control  measure: 

e.  Checkpoints 

f.  Rally  points 

g.  Coordination  points 

h.  Waypoints 

Which  is  not  one  of  the  “5  C’s”? 

e.  Cordon 

f.  Control 

g.  Cover 

h.  Clear 


Mission  planning 

Does  the  student 
understand  basic 
controi  measures? 


Reporting 

Does  the  student 
understand  basic 
reporting 
responsibiiities? 


4.  A  What  are  the  three  main  components  of  the  Situation  paragraph  of 

the  5  paragraph  operational  order? 

a.  Terrain  and  weather;  enemy  situation,  friendly  situation 

b.  Terrain  and  weather;  enemy  situation;  commander’s  intent 

c.  Mission;  concept  of  operations;  commander’s  intent 

d.  Beans;  bullets;  bandaids 

5.  C  Which  of  the  following  should  not  be  done  at  a  short  security  halt? 

e.  Conduct  “5  and  25”  meter  checks 

f.  Maintain  360  degree  security 

g.  Dismounts  establish  a  secure  perimeter 

h.  Man  crew  served  weapons 

6.  A  Which  is  not  a  responsibility  of  the  vehicle  commander? 

e.  Observe  sectors  of  fire 

f.  Provide  direction  to  the  driver 

g.  Communicate  with  the  convoy  commander 

h.  Designate  sectors  of  observation 

7.  C  Who  is  an  appropriate  person  to  conduct  lED  reporting  to  higher 

headquarters? 

e.  Vehicle  commander  of  the  trail  vehicle 

f.  Gunner  of  the  vehicle  who  spotted  the  lED 

g.  Assistant  convoy  commander 

h.  Security  element  commander 


Mission  planning 

OPORD— 
Understanding  of 
basic  OPORD 
structure 


lA  Procedure 

Haits — does 
student 
understand 
difference  betw 
iong  /  short  bait? 


Responsibilities 

Does  the  student 
understand  basic 
crew  tasks? 


Responsibilities  / 
Division  of  labor 

ACC  handies 
administrative  so 
CC  can  fight  the 
unit  /  direct  action 


8.  D  Which  of  the  following  would  you  consider  in  the  “E”  part  of  METT-T? 

a.  Weather  and  its  effects 

b.  Terrain  and  its  effects 

c.  Composition  of  friendly  forces 

d.  Composition  of  enemy  forces 


Mission  planning 

METT-T— 
Understanding  of 
enemy 

considerations  in 
mission  pianning 
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9.  A  Which  is  the  first  step  in  the  recovery  of  a  disabied  vehicie? 

e.  Estabiish  iocai  security  at  the  vehicie  site 

f.  Report  to  higher  headquarters 

g.  Convoy  continues  past  the  vehicie 

h.  Nearest  vehicie  rigs  for  tow  with  strap,  chain,  cabie,  or  tow  bar 


lA  Procedure 

Vehicle  recover — 
does  student 
understand  basic 
execution? 


10.  C  in  the  5  paragraph  operationai  order,  in  which  paragraph  wouid  you 

task  your  subordinate  units? 

a.  Paragraph  1 

b.  Paragraph  2 

c.  Paragraph  3 

d.  Paragraph  4 

1 1 .  A  Which  part  of  the  convoy  typicaiiy  handies  navigation? 

e.  Head 

f.  Security 

g.  Combat  Assauit 

h.  Main  body 

12.  B  Which  troop  ieading  step  aiiows  you  to  gain  information  that  you 

need  to  compiete  your  pian? 

a.  Begin  pianning 

b.  Make  reconnaissance 

c.  Execute  the  operation 

d.  Conduct  rehearsais 


Mission  planning 

OPORD— 
Understanding  of 
basic  OPORD 
structure 


Organizational 
structure  vs  task 
organization 

Does  student 

understand 

organizational? 


Mission  planning 

BAMCIS— 
Understanding  of 
basic  troop 

leading  steps 


13.  D  Which  action  sequence  best  describes  reaction  to  a  sniper? 

e.  Security  eiement  provides  support  by  fire  whiie  main  body 
presses  through  the  kiii  zone 

f.  Security  eiement  provides  support  by  fire  whiie  main  body 
assauits 

g.  Main  body  provides  support  by  fire  whiie  security  eiement 
assauits 

h.  Vehicies  return  suppressive  fire,  speed  up,  and  keep  moving 

14.  C  When  pianning  an  operation,  you  shouid  take _ of  the  time 

avaiiabie  for  yourseif  and  ieave _ of  the  time  avaiiabie  for 

your  subordinates. 


a. 

1/2; 

1/2 

b. 

2/3; 

1/3 

c. 

1/3; 

2/3 

d. 

3/4; 

1/4 

lA  Procedure 

Sniper — does 
student 

understand  basic 
execution? 


Mission  planning 

Understanding  of 
basic  mission 
planning 
fundamentals 


1 5.  C  Which  is  one  of  the  three  organizationai  eiements  of  a  convoy? 

e.  Security  eiement 

f.  Obstacie  Ciearing  Detachment 

g.  Traii 

h.  Assauit  force 


Organizational 
structure  vs  task 
organization 

Does  student 
understand  the 
difference? 


16.  C  if  you  received  a  5  paragraph  operationai  order  from  your  higher 

headquarters,  where  wouid  you  primariiy  iook  to  find  your  mission? 

a.  Friendiy  situation 

b.  Commander’s  intent 

c.  Tasks  to  subordinate  units 

d.  Coordinating  instructions 


Mission  planning 

OPORD— 
Understanding  of 
basic  OPORD 
structure 


272 


17.  C  Who  supervises  recovery  operations  for  the  convoy? 

e.  Vehicle  commander 

f.  Convoy  commander 

g.  Assistant  convoy  commander 

h.  Security  force  commander 


Responsibilities  / 
Division  of  labor 

Does  student 
understand 
delegation  of 
logistic  functions? 


18.  A 


When  analyzing  the  mission,  the  convoy  commander  should 
follow  which  process? 

e.  Mission,  Enemy,  Terrain  and  weather.  Troops  and  fire  support. 
Time  (METT-T) 

f.  Obstacles,  Key  terrain.  Observation  and  fields  of  fire.  Cover  and 
concealment.  Avenues  of  approach  (OKOCA) 

g.  Defend,  Reinforce,  Assault,  Withdraw,  Delay  (DRAW-D) 

h.  Confirm,  Clear,  Cordon,  Check,  Control  (5  C’s) 


Mission  Planning 

Basic 

understanding  of 
mission  planning 
concepts 


19.  C  For  an  unblocked  ambush,  the  element  that  assaults  from  outside 
the  kill  zone  is  the 

e.  Trail 

f.  Main  body 

g.  Security 

h.  Armored  escort 


lA  Procedure 

Ambush — does 
student 

understand  basic 
execution? 


20.  A  When  a  convoy  reacts  to  an  unblocked  ambush,  the  most  critical 
information  to  pass  to  the  convoy  commander  is: 

e.  Direction  of  enemy  fires 

f.  Speed  of  the  lead  vehicle 

g.  Nearest  checkpoint  to  the  lead  vehicle 

h.  Number  of  vehicles  taking  fire 


lA  Procedure 

Unblocked 
ambush — does 
the  student 

understand  basic 
execution? 


21 .  D  In  what  part  of  the  5  paragraph  operational  order  would  you  publish 
the  time  line  for  the  operation? 

a.  Friendly  situation 

b.  Mission  statement 

c.  Tasks  to  subordinate  units 

d.  Coordinating  instructions 


Mission  planning 

OPORD— 
Understanding  of 
basic  OPORD 
structure 


22.  D  The  last  step  in  reacting  to  an  lED  is: 

i.  Cordon 

j.  Report  the  lED  to  higher  headquarters 

k.  Consolidate 

l.  Control 


lA  Procedure 

lED — does 
student 

understand  the  5 
C’s? 


23.  B  In  the  5  paragraph  operational  order,  in  which  paragraph  would  you 
find  the  time  you  should  start  your  mission? 

a.  Paragraph  1 

b.  Paragraph  2 

c.  Paragraph  4 

d.  Paragraph  5 


Mission  planning 

OPORD— 
Understanding  of 
basic  OPORD 
structure 


24.  A  Which  is  an  appropriate  range  for  the  vehicle  interval  in  general 
open  terrain? 

e.  50 -  100  meters 

f.  20  -  50  meters 

g.  200 -400  meters 

h.  150  -  250  meters 


Movement 

Does  the  student 
understand  basic 
convoy  movement 
principles? 
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APPENDIX  Z.  KNOWLEDGE  TEST  B  (EXPERIMENT  5) 


1 .  D  For  a  blocked  ambush,  a  vehicle  that  has  not  yet  entered  the  kill 

zone  should 

a.  Assault 

b.  Continue  to  move 

c.  Speed  up 

d.  Find  cover 

2.  B  The  primary  mission  of  a  convoy  is: 

e.  Route  clearance 

f.  Moving  personnel  and  /  or  cargo 

g.  Intelligence  collection 

h.  Destroying  enemy  combatants 

3.  D  When  reacting  to  an  lED,  the  convoy  commander  should  follow 

which  process? 

e.  Mission,  Enemy,  Terrain  and  weather.  Troops  and  fire  support. 
Time  (METT-T) 

f.  5  /  25’s 

g.  Defend,  Reinforce,  Assault,  Withdraw,  Delay  (DRAW-D) 

h.  Confirm,  Clear,  Call,  Cordon,  Control  (5  C’s) 

4.  B  When  considering  the  effects  of  terrain,  which  is  not  one  of  the 

standard  considerations  as  described  by  the  acronym  OCOKA? 

a.  Observation  and  fields  of  fire 

b.  Orientation 

c.  Cover  and  concealment 

d.  Avenues  of  approach 

5.  B  Which  of  the  following  is  a  way  that  short  and  long  security  halts 

are  the  same? 

e.  Amount  of  dismounted  support  employed 

f.  Vehicle  crew  served  weapons  are  manned  and  oriented  on 
designated  sector  of  fire 

g.  Amount  of  cover  employed  by  dismounts 

h.  Vulnerability  of  main  body 

6.  D  Where  in  the  convoy  should  the  convoy  commander  place  himself? 

e.  Lead  vehicle 

f.  Trail  vehicle 

g.  With  the  security  element  commander 

h.  Main  body 

7.  C  Who  is  an  appropriate  person  to  conduct  casualty  reporting? 

e.  Combat  life  saver 

f.  Corpsman 

g.  Assistant  convoy  commander 

h.  Security  element  commander 

8.  D  Which  of  the  following  correctly  represents  the  “T’s”  in  METT-T? 

a.  Training,  Troops,  Time 

b.  Terrain,  Training,  Time 

c.  Terrain,  Trucks,  Time 

d.  Terrain,  Troops,  Time 


lA  Procedure 

Unblocked 
ambush — does 
student 

understand  basic 
execution? 


Mission  planning 

Does  student 
understand  the 
purpose  of 

convoy? 


lA  Procedure 

iED — does  the 

student 

understand  the  5 
C’s? 


Mission  planning 

METT-T— 
Understanding  of 
terrain  anaiysis 
steps 


lA  Procedure 

Haits — crew 
served  weapons 
must  aiways  be 
empioyed  in 
security  as  priority 


Responsibilities 

Does  the  student 
understand  CC 
enough  not  to  be 
in  front  iOT 
maintain  controi? 


Responsibilities  / 
Division  of  labor 

Does  student 
understand  to 
delegate  reporting 
so  CC  can  fight? 


Mission  planning 

METT-T— 
Understanding  of 
basic  mission 
planning  steps 
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9.  D  When  a  convoy  vehicle  first  becomes  disabled,  the  security  element 
commander  should  consider  the  following  as  his  primary  responsibility: 

e.  Reporting  the  disabled  vehicle  to  higher  headquarters 

f.  Punching  forward  to  determine  whether  enemy  are  waiting 
to  ambush 

g.  Ensuring  the  convoy  stops  in  a  herringbone  in  case  of  air 
attack 

h.  Providing  security  for  the  convoy  main  body 


lA  Procedure 

Vehicle  rec — does 
the  student 

understand  to 
maintain  convoy 
security  first? 


10.  D  In  the  5  paragraph  operational  order,  in  which  paragraph  would 

you  find  the  frequency  of  the  higher  command  radio  net? 

a.  Paragraph  2 

b.  Paragraph  3 

c.  Paragraph  4 

d.  Paragraph  5 

1 1 .  D  Who  specifies  the  speed  of  the  convoy? 

e.  Lead  vehicle  commander 

f.  Security  element  commander 

g.  Driver  of  the  lead  vehicle 

h.  Convoy  Commander 


Mission  planning 

OPORD— 
Understanding  of 
basic  OPORD 
structure 


Responsibilities 

Does  the  student 
understand  that 
the  logistics  in 
main  body 

determines  spd? 


12.  C  Which  of  the  following  is  not  one  of  the  troop  leading  steps? 

a.  Supervise 

b.  Complete  the  plan 

c.  Execute  the  operation 

d.  Begin  planning 

13.  A  Which  action  should  the  convoy  commander  avoid  when 

reacting  to  a  sniper? 

e.  Stop 

f.  Speed  up 

g.  Return  fire 

h.  Use  smoke 

14.  B  In  order  to  maintain  flexibility  in  the  face  of  a  potential  threat,  a 

convoy  commander  should: 

e.  Focus  on  speed  and  momentum  throughout  the  convoy 

f.  Plan  an  alternate  route 

g.  Place  himself  in  the  lead  of  the  convoy 

h.  Place  himself  in  the  trail  of  the  convoy 

1 5.  B  Which  is  one  of  the  three  elements  of  a  convoy  task  organization? 

e.  Trail 

f.  Security 

g.  Main  body 

h.  Head 


Mission  planning 

BAMCIS— 
Understanding  of 
basic  troop 

leading  steps 


lA  Procedure 

Sniper — does  the 
student 

understand  basic 
execution? 


Mission  planning 

Does  the  student 
understand  how 
mission  planning 
affects  future 
execution? 


Organizational 
structure  vs  task 
organization 

Does  the  student 
understand  task 
organization? 


1 6.  D  What  part  of  the  operational  order  would  you  use  to  ensure  your 
subordinates  understood  the  final  end  state  of  the  operation? 

a.  Concept  of  operations 

b.  Coordinating  instructions 

c.  Command  and  signal 

d.  Commander’s  intent 


Mission  planning 

OPORD— 
Understanding  of 
basic  OPORD 
structure 


276 


17.  B  Who  maintains  communication  with  higher  and  adjacent 
authorities  for  the  convoy? 

e.  Vehicle  commander 

f.  Convoy  commander 

g.  Assistant  convoy  commander 

h.  Security  force  commander 


Responsibilities  / 
Division  of  labor 

Does  student 
understand  to 
delegate  reporting 
and  CC  fight? 


18.  B  Why  would  you  use  METT-T? 

a.  Describe  the  effects  of  terrain  on  the  operation 

b.  Analyze  your  mission  in  order  to  plan  the  operation 

c.  Develop  the  time  line  for  the  operation 

d.  Coordinate  indirect  and  direct  fire  plans  for  the  concept 
of  operations 

1 9.  B  It  would  be  inappropriate  to  speed  up  for  which  immediate 

action  drill? 

e.  Reaction  to  sniper 

f.  Reaction  to  unexploded  lED 

g.  Reaction  to  unblocked  ambush 

h.  All  of  the  above 


Mission  planning 

METT-T— 
Understanding  the 
purpose  of  METT- 
T  as  an  aid  to 
mission  planning 


lA  Procedure 

Ambush — does 
the  student 

understand  basic 
execution? 


20.  B  When  crossing  a  danger  area,  which  element  provides  over 
watch  while  the  main  body  crosses? 

e.  Obstacle  Clearing  Detachment 

f.  Trail 

g.  Main  body’s  security  elements 

h.  Combat  Assault  Element 


lA  Procedure 

Danger  area — 
does  student 

understand  basic 
execution 


21 .  B  Which  part  of  the  5  paragraph  operational  order  paints  a  word 
picture  of  how  the  operation  should  unfold? 

a.  Terrain  and  weather 

b.  Concept  of  operations 

c.  Friendly  situation 

d.  Mission 


Mission  planning 

OPORD— 
Understanding  of 
basic  OPORD 
structure 


22.  D  The  first  step  in  reacting  to  an  lED  is: 

a.  Cordon 

b.  Report  the  lED  to  higher  headquarters 

c.  Call  EOD 

d.  Confirm 


lA  Procedure 

lED — does  the 

student 

understand  the  5 
C’s? 


23.  A  In  the  5  paragraph  operational  order,  in  which  paragraph  would  you 
find  the  enemy’s  most  likely  course  of  action? 

a.  Paragraph  1 

b.  Paragraph  2 

c.  Paragraph  3 

d.  Paragraph  4 


Mission  planning 

OPORD— 
Understanding  of 
basic  OPORD 
structure 


24.  B 


Which  of  the  following  should  not  be  used  to  determine  convoy  speed? 

e.  Terrain 

f.  Need  for  crew  rest 

g.  Weather  conditions 

h.  Likelihood  of  enemy  contact 


Movement 

Does  the  student 
understand  basic 
movement 
principies? 
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